First transfer¶

In this step you transfer a single file from GitHub (HTTP) into the SeaweedFS bucket from the setup. All files are in examples/walk-through/01-first-transfer.

Five resources are involved:

Resource	Owned by	Purpose
`Secret`	you	Credentials for the destination
`Endpoint`	ARC	Where to pull from and push to
`WorkflowTemplate`	Argo	How to transfer, the actual work
`ArtifactType`	ARC	Connects an artifact type to a `WorkflowTemplate` and allowed endpoints
`Order`	ARC	What to transfer

Secret¶

The destination needs credentials. The keys of a Secret are up to you, the WorkflowTemplate is what reads them.

# Credentials for the SeaweedFS destination from the setup step.
apiVersion: v1
kind: Secret
metadata:
  name: seaweedfs-credentials
type: Opaque
stringData:
  AWS_ACCESS_KEY_ID: admin
  AWS_SECRET_ACCESS_KEY: adminadmin

kubectl apply -f examples/walk-through/01-first-transfer/1-secret.yaml

Endpoints¶

An Endpoint describes a remote location. Its type is free-form and is used to decide which artifact types may use it. usage restricts it to pulling, pushing or both, and secretRef points to the credentials.

apiVersion: arc.opendefense.cloud/v1alpha1
kind: Endpoint
metadata:
  name: github-raw
spec:
  type: http # Endpoint type, matched against ArtifactType rules
  remoteURL: https://raw.githubusercontent.com
  usage: PullOnly
  # no secretRef: public source
---
apiVersion: arc.opendefense.cloud/v1alpha1
kind: Endpoint
metadata:
  name: seaweedfs
spec:
  type: s3
  remoteURL: http://dst.seaweedfs:8333
  usage: PushOnly
  secretRef:
    name: seaweedfs-credentials

kubectl apply -f examples/walk-through/01-first-transfer/2-endpoints.yaml

WorkflowTemplate¶

This is where the work happens. It is a plain Argo WorkflowTemplate with a single script template that downloads the file with curl and streams it into the bucket with the AWS CLI.

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: blob-transfer
spec:
  serviceAccountName: arc-workflow
  # ARC passes these parameters to every workflow it creates. All values are strings.
  arguments:
    parameters:
      # from the source Endpoint
      - name: srcType
      - name: srcRemoteURL
      - name: srcSecret # "true" if the Endpoint has a secretRef, otherwise "false"
      # from the destination Endpoint
      - name: dstType
      - name: dstRemoteURL
      - name: dstSecret
      # from the artifact spec in the Order: spec.key -> specKey, spec.override -> specOverride
      - name: specKey
      - name: specOverride
      # Also passed by ARC, but not used by this template:
      # - name: cron # "true" if the artifact is scheduled by a cron, otherwise "false"
      # - name: <name> # every parameter of the ArtifactType, overriding the ones above
  entrypoint: transfer
  templates:
    - name: transfer
      script:
        image: amazon/aws-cli:2.37.9
        command: [bash]
        volumeMounts:
          # ARC adds the volumes src-secret-vol and dst-secret-vol to every workflow.
          # The template decides where (and whether) to mount them.
          - name: dst-secret-vol
            mountPath: /secrets/dst
        env: # never put parameters into the source, a value could run as code
          - name: SPEC_KEY
            value: "{{workflow.parameters.specKey}}"
          - name: SPEC_OVERRIDE
            value: "{{workflow.parameters.specOverride}}"
          - name: SRC_REMOTE_URL
            value: "{{workflow.parameters.srcRemoteURL}}"
          - name: DST_REMOTE_URL
            value: "{{workflow.parameters.dstRemoteURL}}"
        source: |
          set -euo pipefail
          export AWS_ACCESS_KEY_ID=$(cat /secrets/dst/AWS_ACCESS_KEY_ID)
          export AWS_SECRET_ACCESS_KEY=$(cat /secrets/dst/AWS_SECRET_ACCESS_KEY)

          # SUPPLY CHAIN: this copies the blob without inspecting it. Never do this in
          # production: verify it (checksum, virus scan, SBOM, ...) before it reaches
          # the destination. The next step of the walk-through (02-verify) adds a
          # checksum and a virus scan.
          echo "Transferring ${SRC_REMOTE_URL}/${SPEC_KEY}"
          echo "          to ${DST_REMOTE_URL}/${SPEC_OVERRIDE}"
          curl -fsSL "${SRC_REMOTE_URL}/${SPEC_KEY}" \
            | aws --endpoint-url "${DST_REMOTE_URL}" \
                s3 cp --no-progress - "s3://${SPEC_OVERRIDE}"
          echo "Done."

kubectl apply -f examples/walk-through/01-first-transfer/3-workflow-template.yaml

ARC provides two things to every workflow it creates:

Parameters, accessible as {{workflow.parameters.<name>}} (see workflow variables). They describe both endpoints and the artifact. Every field of the artifact's spec in the Order becomes a parameter prefixed with spec, so key turns into specKey. See Workflow Parameters for all the rules.
Volumes named src-secret-vol and dst-secret-vol containing the endpoint secrets. If an Endpoint has no secret, an empty volume is used instead. The template decides where to mount them, here the destination credentials end up in /secrets/dst.

Parameters never contain secrets. Use srcSecret and dstSecret to find out whether credentials were provided.

Parameters are data, not code

Argo replaces {{workflow.parameters.<name>}} in the script source before bash runs it, so a value from an Order would run as code, with the credentials of the step mounted. Pass parameters to scripts as environment variables with env, like this template does, and quote them in the script, e.g. "${SPEC_KEY}". List the spec* parameters first: Kubernetes expands $(NAME) in an env value with the variables listed before it.

Supply chain security

This workflow copies the file into the destination without looking at it. Never do this in production: anything you transfer, you also distribute. Verify artifacts before pushing them, e.g. by checking a checksum, scanning for viruses or creating an SBOM. The next step adds a checksum and a virus scan.

ArtifactType¶

The ArtifactType named blob connects the WorkflowTemplate with orders. Its rules define which endpoint types are allowed as source and destination. An Order for a blob from e.g. an oci endpoint is still accepted by the API, but ARC creates no workflow for it and reports the reason as a warning event, visible with kubectl describe orders.arc.opendefense.cloud <name>.

apiVersion: arc.opendefense.cloud/v1alpha1
kind: ArtifactType
metadata:
  name: blob
spec:
  rules:
    srcTypes: # allowed Endpoint types
      - http
    dstTypes:
      - s3
  workflowTemplateRef:
    name: blob-transfer # namespaced WorkflowTemplate (clusterScope defaults to false)
  # Keep succeeded workflows (and their logs) for an hour. If unset, ARC deletes them
  # right after they succeed.
  ttlAfterFinished: 1h

By default, ARC deletes a workflow as soon as it succeeded, failed ones are kept for troubleshooting. ttlAfterFinished keeps succeeded workflows around for a while, so you can inspect them and read their logs. See TTL-Based Cleanup for all options.

kubectl apply -f examples/walk-through/01-first-transfer/4-artifact-type.yaml

Note

An ArtifactType only applies to its namespace. It references a namespaced WorkflowTemplate by default, or a ClusterWorkflowTemplate with workflowTemplateRef.clusterScope: true. Use a ClusterArtifactType to offer an artifact type to all namespaces. If both exist with the same name, the namespaced ArtifactType wins.

See Core Concepts for how the ARC resources relate to each other and Extending ArtifactTypes for more on writing your own.

Order¶

Finally, the Order describes what to transfer. defaults sets the endpoints for all artifacts, each artifact can override them with its own srcRef and dstRef. The spec is free-form and only interpreted by the workflow.

apiVersion: arc.opendefense.cloud/v1alpha1
kind: Order
metadata:
  name: first-transfer
spec:
  defaults:
    srcRef:
      name: github-raw
    dstRef:
      name: seaweedfs
  artifacts:
    - type: blob # ArtifactType
      spec: # free-form, handed to the workflow as spec* parameters
        key: opendefensecloud/artifact-conduit/refs/tags/v0.3.0/examples/blob/file.txt
        override: test/walk-through/file.txt # <bucket>/<key> in the destination

Apply it last: the Order immediately creates a workflow, which fails if its WorkflowTemplate does not exist yet.

kubectl apply -f examples/walk-through/01-first-transfer/5-order.yaml

Tip

The files are numbered, so kubectl apply -f examples/walk-through/01-first-transfer/ applies all of them in the right order. If an Order was applied too early, a failed workflow is not retried on its own. Fix the cause and force a reconciliation to recreate it.

Watch it run¶

For every artifact in the Order, ARC creates an ArtifactWorkflow, which in turn creates an Argo Workflow:

kubectl get artifactworkflows
kubectl get workflows

Wait for the workflow to finish and read its logs:

kubectlArgo CLI

kubectl wait workflows --all --for jsonpath='{.status.phase}'=Succeeded --timeout 5m
kubectl logs -l workflows.argoproj.io/workflow -c main --tail -1

Lines with argo=true come from the Argo executor that runs the script, not from the script itself.

argo watch @latest
argo logs @latest

Once the workflow succeeded, the Order reports it in its status:

kubectl get orders.arc.opendefense.cloud first-transfer -o jsonpath='{.status.artifactWorkflows}'

Note

cert-manager also has a resource called Order, so plain kubectl get order may return cert-manager's. The fully qualified name orders.arc.opendefense.cloud always works.

Check that the file arrived in the bucket:

kubectl run s3-check --rm -it --restart Never --image amazon/aws-cli:2.37.9 \
  --env AWS_ACCESS_KEY_ID=admin --env AWS_SECRET_ACCESS_KEY=adminadmin -- \
  --endpoint-url http://dst.seaweedfs:8333 s3 cp s3://test/walk-through/file.txt -

If something does not work, have a look at Debugging.

Next: verify the file before you push it.