Skip to content

Setup

This step prepares a local cluster for the walk-through. If you already have a cluster with ARC and Argo Workflows, you can skip to creating the namespace.

You need git, kubectl, helm and a Kubernetes cluster, e.g. one created with kind. From the verify step on, workflows run a ClamAV virus scan. It pulls a large image and loads its virus database into memory, so give a local cluster enough resources.

All files used in the walk-through live in the ARC repository, so clone it first:

git clone https://github.com/opendefensecloud/artifact-conduit.git
cd artifact-conduit

Create a cluster

Any Kubernetes cluster works. For a local one, use kind:

kind create cluster --name arc-walk-through

Install Argo Workflows

ARC creates Argo Workflows, so Argo Workflows has to be installed. See the Argo Workflows installation for details. The short version, using the Argo Workflows version ARC is built and tested with:

export ARGO_WORKFLOWS_VERSION="$(awk '/^[ \t]+github.com\/argoproj\/argo-workflows/ {print $2}' go.mod)"
kubectl create namespace argo
kubectl apply --server-side -n argo -f "https://github.com/argoproj/argo-workflows/releases/download/${ARGO_WORKFLOWS_VERSION}/install.yaml"
kubectl wait deployment/workflow-controller -n argo --for condition=Available --timeout 5m

Optionally install the Argo CLI. It is not required, but makes it easier to follow workflows and read their logs. The walk-through shows commands for both the Argo CLI and plain kubectl, pick a tab once and all pages follow.

Install ARC

ARC needs cert-manager for the certificates of its API server. Install the version ARC is tested with:

export CERTMANAGER_VERSION="$(awk '/^export CERTMANAGER_VERSION/ {print $NF}' Makefile)"
kubectl apply -f "https://github.com/cert-manager/cert-manager/releases/download/${CERTMANAGER_VERSION}/cert-manager.yaml"
kubectl wait deployment -n cert-manager --all --for condition=Available --timeout 5m

Then install ARC with the release your checkout is based on (see the Quick Start for other ways to pick a version):

export ARC_VERSION="$(git describe --tags --abbrev=0)"
helm install arc oci://ghcr.io/opendefensecloud/charts/arc \
  --version "${ARC_VERSION}" \
  --namespace arc-system \
  --create-namespace \
  --set fullnameOverride=arc \
  --set apiserver.args.cronMinScheduleInterval=1m

Note

The walk-through is tested against ARC built from the same commit, not against a release. On main, a step may use a feature that is not released yet and fail with the chart above. In that case, check out the release you installed with git checkout "${ARC_VERSION}" and follow the docs of that version.

By default, ARC rejects schedules that run more often than every 10 minutes. The walk-through lowers that to one minute, so you do not have to wait long when you schedule transfers. See Cron Minimal Schedule Interval.

ARC's API is served by an extension API server. Wait until it is available, otherwise kubectl does not know the ARC resources yet:

kubectl wait apiservice/v1alpha1.arc.opendefense.cloud --for condition=Available --timeout 5m

Create a namespace for the walk-through

Everything in the walk-through is namespaced. Create a namespace and make it the default for kubectl:

kubectl create namespace walk-through
kubectl config set-context --current --namespace walk-through

Create a ServiceAccount for workflows

Pods of an Argo Workflow run under a ServiceAccount that needs a few permissions, e.g. to report step results back to Argo. See Workflow RBAC for details. The workflows in this walk-through use a ServiceAccount named arc-workflow:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: arc-workflow
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: arc-workflow
rules:
  - apiGroups:
      - argoproj.io
    resources:
      - workflowtaskresults
    verbs:
      - create
      - patch
  - apiGroups:
      - ""
    resources:
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - persistentvolumes
      - persistentvolumeclaims
      - persistentvolumeclaims/status
    verbs:
      - create
      - delete
      - get
      - list
      - watch
      - update
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: arc-workflow
subjects:
  - kind: ServiceAccount
    name: arc-workflow
roleRef:
  kind: Role
  name: arc-workflow
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f test/fixtures/service-account.yaml

Install an S3 store as destination

The walk-through transfers files into an S3 bucket. SeaweedFS provides a small S3-compatible store with a bucket named test:

# Single-process SeaweedFS (weed mini) as S3 destination. It creates an admin
# identity from AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY (must match the Secret of
# the Endpoint pointing here) and the bucket S3_BUCKET on startup.
apiVersion: v1
kind: Namespace
metadata:
  name: seaweedfs
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dst
  namespace: seaweedfs
spec:
  selector:
    matchLabels:
      app: dst
  template:
    metadata:
      labels:
        app: dst
    spec:
      containers:
        - name: seaweedfs
          image: chrislusf/seaweedfs:4.48
          env:
            - name: AWS_ACCESS_KEY_ID
              value: admin
            - name: AWS_SECRET_ACCESS_KEY
              value: adminadmin
            - name: S3_BUCKET
              value: test
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8333
---
apiVersion: v1
kind: Service
metadata:
  name: dst
  namespace: seaweedfs
spec:
  selector:
    app: dst
  ports:
    - port: 8333

kubectl apply -f test/fixtures/dst-seaweedfs.yaml
kubectl wait deployment/dst -n seaweedfs --for condition=Available --timeout 5m

You are ready for your first transfer.

Clean up

When you are done with the walk-through, delete the cluster:

kind delete cluster --name arc-walk-through