Conditionals

So far, every file came from GitHub over HTTP. In this step the workflow also pulls from S3 and picks the right way to pull for each artifact with an Argo conditional. One Order transfers one file from each source. All files are in examples/walk-through/03-conditionals.

Endpoint

The S3 source is the SeaweedFS from the setup again, this time as a PullOnly endpoint. Unlike github-raw, it needs credentials and reuses the Secret of the first transfer.

# The SeaweedFS from the setup again, this time as a source. An Endpoint is
# either PullOnly, PushOnly or All. A second Endpoint keeps "seaweedfs" PushOnly.
apiVersion: arc.opendefense.cloud/v1alpha1
kind: Endpoint
metadata:
  name: seaweedfs-source
spec:
  type: s3
  remoteURL: http://dst.seaweedfs:8333
  usage: PullOnly
  secretRef:
    name: seaweedfs-credentials # the same credentials as for pushing

kubectl apply -f examples/walk-through/03-conditionals/1-endpoint.yaml

WorkflowTemplate

The new blob-transfer has two pull steps instead of one, pull-http and pull-s3. Both are in the same group of steps, but each has a when condition on the srcType parameter, so only one runs and the other is skipped. The checks and push stay as they are, they do not care where the file came from.

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: blob-transfer # replaces the template of the previous step
spec:
  serviceAccountName: arc-workflow
  # ARC passes these parameters to every workflow it creates. All values are strings.
  arguments:
    parameters:
      # from the source Endpoint
      - name: srcType
      - name: srcRemoteURL
      - name: srcSecret # "true" if the Endpoint has a secretRef, otherwise "false"
      # from the destination Endpoint
      - name: dstType
      - name: dstRemoteURL
      - name: dstSecret
      # from the artifact spec in the Order: spec.key -> specKey, ...
      - name: specKey
      - name: specOverride
      - name: specSha256 # no default: orders without a checksum fail
      # Also passed by ARC, but not used by this template:
      # - name: cron # "true" if the artifact is scheduled by a cron, otherwise "false"
      # - name: <name> # every parameter of the ArtifactType, overriding the ones above
  # Steps run in pods of their own, so they share files through a volume. Argo creates
  # this PersistentVolumeClaim for every workflow and deletes it when the workflow is done.
  # The size is fixed here, examples/blob sizes the volume from the blob instead.
  volumeClaimTemplates:
    - metadata:
        name: work
      spec:
        accessModes: [ReadWriteOnce]
        resources:
          requests:
            storage: 1Gi
  entrypoint: transfer
  templates:
    # Every step runs after the previous one succeeded. If a check fails, the
    # workflow stops and the blob never reaches the destination.
    - name: transfer
      steps:
        # Both pull steps are in the same group, but only the one whose condition
        # is true runs, the other one is skipped. srcType comes from the Endpoint, not
        # the Order. Don't put spec* parameters into a condition, they could change it.
        - - name: pull-http
            template: pull-http
            when: "'{{workflow.parameters.srcType}}' == 'http'"
          - name: pull-s3
            template: pull-s3
            when: "'{{workflow.parameters.srcType}}' == 's3'"
        - - name: checksum
            template: checksum
        - - name: virus-scan
            template: virus-scan
        - - name: push
            template: push

    - name: pull-http
      script:
        image: amazon/aws-cli:2.37.9
        command: [bash]
        volumeMounts:
          - name: work
            mountPath: /work
        env: # parameters as environment variables, see step 1
          - name: SPEC_KEY
            value: "{{workflow.parameters.specKey}}"
          - name: SRC_REMOTE_URL
            value: "{{workflow.parameters.srcRemoteURL}}"
        source: |
          set -euo pipefail
          echo "Pulling ${SRC_REMOTE_URL}/${SPEC_KEY}"
          curl -fsSL -o /work/blob "${SRC_REMOTE_URL}/${SPEC_KEY}"

    - name: pull-s3
      script:
        image: amazon/aws-cli:2.37.9
        command: [bash]
        volumeMounts:
          - name: work
            mountPath: /work
          - name: src-secret-vol
            mountPath: /secrets/src
        env:
          - name: SPEC_KEY
            value: "{{workflow.parameters.specKey}}"
          - name: SRC_SECRET
            value: "{{workflow.parameters.srcSecret}}"
          - name: SRC_REMOTE_URL
            value: "{{workflow.parameters.srcRemoteURL}}"
        source: |
          set -euo pipefail
          # Credentials are optional: public buckets are read without signing requests.
          AUTH=(--no-sign-request)
          if [ "${SRC_SECRET}" = "true" ]; then
            export AWS_ACCESS_KEY_ID=$(cat /secrets/src/AWS_ACCESS_KEY_ID)
            export AWS_SECRET_ACCESS_KEY=$(cat /secrets/src/AWS_SECRET_ACCESS_KEY)
            AUTH=()
          fi
          echo "Pulling ${SRC_REMOTE_URL}/${SPEC_KEY}"
          aws "${AUTH[@]}" --endpoint-url "${SRC_REMOTE_URL}" \
            s3 cp --no-progress "s3://${SPEC_KEY}" /work/blob

    - name: checksum
      script:
        image: amazon/aws-cli:2.37.9
        command: [bash]
        volumeMounts:
          - name: work
            mountPath: /work
            readOnly: true # checks must not change what they check
        env:
          - name: SPEC_SHA256
            value: "{{workflow.parameters.specSha256}}"
        source: |
          set -euo pipefail
          echo "${SPEC_SHA256}  /work/blob" | sha256sum --check

    - name: virus-scan
      script:
        image: clamav/clamav-debian:1.5.4 # ships with a virus database
        command: [bash]
        volumeMounts:
          - name: work
            mountPath: /work
            readOnly: true
        source: |
          set -euo pipefail
          clamscan /work/blob

    # More checks go here as further steps, e.g. creating an SBOM or verifying a signature.

    - name: push
      script:
        image: amazon/aws-cli:2.37.9
        command: [bash]
        volumeMounts:
          - name: work
            mountPath: /work
          - name: dst-secret-vol
            mountPath: /secrets/dst
        env:
          - name: SPEC_OVERRIDE
            value: "{{workflow.parameters.specOverride}}"
          - name: DST_REMOTE_URL
            value: "{{workflow.parameters.dstRemoteURL}}"
        source: |
          set -euo pipefail
          export AWS_ACCESS_KEY_ID=$(cat /secrets/dst/AWS_ACCESS_KEY_ID)
          export AWS_SECRET_ACCESS_KEY=$(cat /secrets/dst/AWS_SECRET_ACCESS_KEY)
          echo "Pushing to ${DST_REMOTE_URL}/${SPEC_OVERRIDE}"
          aws --endpoint-url "${DST_REMOTE_URL}" \
            s3 cp --no-progress /work/blob "s3://${SPEC_OVERRIDE}"
          echo "Done."

kubectl apply -f examples/walk-through/03-conditionals/2-workflow-template.yaml

pull-s3 shows the second kind of conditional: whether the source has credentials is decided inside the script, with the srcSecret parameter. src-secret-vol is always there, but it is an empty volume if the Endpoint has no secretRef. See Conditionals for the syntax of when and Enhanced Depends Logic for conditions on the results of other steps.

ArtifactType

The rules of the ArtifactType only allowed http sources. Add s3, otherwise ARC creates no workflow for an artifact from seaweedfs-source:

# The spec of an ArtifactType is immutable: change it with "kubectl replace --force".
apiVersion: arc.opendefense.cloud/v1alpha1
kind: ArtifactType
metadata:
  name: blob
spec:
  rules:
    srcTypes:
      - http
      - s3 # new: the WorkflowTemplate can now pull from S3
    dstTypes:
      - s3
  workflowTemplateRef:
    name: blob-transfer
  ttlAfterFinished: 1h

The spec of an ArtifactType is immutable, kubectl apply refuses to change it. Replace it instead, --force deletes the ArtifactType and creates it again. Workflows that already exist are not affected.

kubectl replace --force -f examples/walk-through/03-conditionals/3-artifact-type.yaml

Order

The Order has two artifacts. The first one uses the default source github-raw. The second one sets its own srcRef and pulls the file that the previous step pushed. Since seaweedfs-source has type: s3, its workflow gets srcType=s3 and runs pull-s3.

apiVersion: arc.opendefense.cloud/v1alpha1
kind: Order
metadata:
  name: multi-source-transfer
  labels:
    walk-through: conditionals
spec:
  defaults:
    srcRef:
      name: github-raw
    dstRef:
      name: seaweedfs
  # One workflow per artifact.
  artifacts:
    - type: blob # uses the default source: github-raw
      spec:
        key: opendefensecloud/artifact-conduit/refs/tags/v0.3.0/examples/blob/file.txt
        override: test/walk-through/from-http.txt
        sha256: "b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9"
    - type: blob
      srcRef:
        name: seaweedfs-source # overrides the default source for this artifact
      spec:
        key: test/walk-through/verified.txt # <bucket>/<key>, pushed in the previous step
        override: test/walk-through/from-s3.txt
        sha256: "b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9"

kubectl apply -f examples/walk-through/03-conditionals/4-order.yaml

Watch it run

ARC creates one ArtifactWorkflow and one Argo Workflow per artifact, so there are two this time:

kubectl get workflows -l walk-through=conditionals

kubectlArgo CLI

kubectl wait workflows -l walk-through=conditionals --for jsonpath='{.status.phase}'=Succeeded --timeout 5m
kubectl get workflows -l walk-through=conditionals \
  -o jsonpath='{range .items[*]}{range .status.nodes.*}{.name}{": "}{.phase}{"\n"}{end}{end}' | sort

argo list -l walk-through=conditionals
argo get <name>

argo get shows the steps of a workflow as a tree, skipped steps included.

In each workflow, one of pull-http and pull-s3 is Skipped. Besides the steps, the list contains the workflow itself and the groups of steps, [0] to [3]. Check that both files arrived in the bucket:

kubectl run s3-check --rm -it --restart Never --image amazon/aws-cli:2.37.9 \
  --env AWS_ACCESS_KEY_ID=admin --env AWS_SECRET_ACCESS_KEY=adminadmin -- \
  --endpoint-url http://dst.seaweedfs:8333 s3 ls s3://test/walk-through/

Next: schedule transfers.