Caching for Kubernetes workers

Kubernetes worker run pods can share volumes for both tool binaries and Terraform/OpenTofu providers. See Caching for an explanation of both caches, their storage requirements and their limitations.

Choose a shared volume

The cache volume must meet the shared storage requirements and provide a ReadWriteMany claim with volumeMode: Filesystem.

Managed options include Amazon EFS, Amazon FSx for OpenZFS, Azure Files and Google Cloud Filestore. Each has a CSI driver that can provision ReadWriteMany claims. Amazon EFS is not recommended because its per-operation latency makes it too slow for caches containing many small files. FSx for OpenZFS has lower latency but comes with minimum filesystem size and throughput costs.

Block StorageClasses cannot provide the required filesystem directly. For example, the EBS CSI driver rejects a ReadWriteMany claim with volumeMode: Filesystem and reports Volume capability not supported. To use a block volume, run an NFS server in the cluster and export the volume through it. The example below shows how to do this on Amazon EKS.

Configure the binaries cache

Set spec.pod.binariesCacheVolume to a Kubernetes volume definition that references a ReadWriteMany claim. The controller mounts the volume at /opt/spacelift/binaries_cache in the run pod. No other configuration is needed.

apiVersion: workers.spacelift.io/v1beta1
kind: WorkerPool
metadata:
  name: worker-pool
spec:
  poolSize: 2
  privateKey:
    secretKeyRef:
      key: privateKey
      name: pool-credentials
  token:
    secretKeyRef:
      key: token
      name: pool-credentials
  pod:
    binariesCacheVolume:
      name: binaries-cache
      persistentVolumeClaim:
        claimName: binaries-cache

Configure the provider cache

Terraform and OpenTofu store downloaded providers in the directory set by the TF_PLUGIN_CACHE_DIR environment variable. Later runs can reuse the providers instead of downloading them again. Because init runs in the run pod's worker container, the cache must be mounted in that container. Add a volume to the pod and mount it into the worker container:

spec:
  pod:
    volumes:
      - name: provider-cache
        persistentVolumeClaim:
          claimName: provider-cache
    workerContainer:
      volumeMounts:
        - name: provider-cache
          mountPath: /opt/spacelift/provider-cache
      env:
        - name: TF_PLUGIN_CACHE_DIR
          value: /opt/spacelift/provider-cache

Setting TF_PLUGIN_CACHE_DIR on the worker container enables the cache for every stack using the pool. To enable it for selected stacks only, leave the env entry out and set TF_PLUGIN_CACHE_DIR as an environment variable on those stacks instead, pointing at the same mount path.

Use a separate claim for the provider cache instead of a subdirectory in the binaries cache. The caches grow at different rates, and separate claims let you clear them independently.

Example: NFS server backed by EBS on Amazon EKS

The kubernetes-sigs NFS Ganesha server and external provisioner runs a single NFS server pod backed by a PersistentVolumeClaim. It provides a StorageClass that creates ReadWriteMany volumes as exported subdirectories. You can use any ReadWriteOnce volume for the backing claim. This setup stores the caches on a regular EBS volume and shares them with run pods over NFS.

The steps below assume an EKS cluster with the EBS CSI driver installed and a gp3 StorageClass named ebs-sc. You can use the same approach with any ReadWriteOnce StorageClass. The nodes also need an NFS client. The Amazon EKS optimized Amazon Linux 2023 AMI includes nfs-utils, so it does not require any node preparation.

Install the NFS server

helm repo add nfs-ganesha https://kubernetes-sigs.github.io/nfs-ganesha-server-and-external-provisioner
helm repo update

helm upgrade --install spacelift-cache-nfs nfs-ganesha/nfs-server-provisioner \
  --namespace spacelift-cache --create-namespace \
  --set persistence.enabled=true \
  --set persistence.storageClass=ebs-sc \
  --set persistence.size=50Gi \
  --set storageClass.name=spacelift-cache-nfs \
  --set 'storageClass.mountOptions={vers=4.1,retrans=2,timeo=30}' \
  --set resources.requests.cpu=100m \
  --set resources.requests.memory=128Mi

The command creates a StatefulSet with one nfs-server-provisioner pod, a 50Gi EBS-backed claim and a StorageClass named spacelift-cache-nfs. The chart mounts NFSv3 by default. The vers=4.1 setting uses NFSv4.1, which only needs one port and provides better locking.

Create the cache claims

Create one claim for each cache on the new StorageClass:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: binaries-cache
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: spacelift-cache-nfs
  resources:
    requests:
      storage: 15Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: provider-cache
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: spacelift-cache-nfs
  resources:
    requests:
      storage: 30Gi

Allow space for filesystem overhead

The provisioner checks the available space on the exported filesystem, which all claims on the StorageClass share. A 20Gi disk has slightly less than 20Gi free after formatting. As a result, a 20Gi claim on a 20Gi backing volume stays Pending, and the provisioner logs insufficient available space. Size the backing volume to hold all claims plus filesystem overhead.

Create the claims in the same namespace as your WorkerPool, then reference them as shown in Configure the binaries cache and Configure the provider cache.

Verify

Run pods are removed as soon as they finish. To inspect a cache, create a temporary pod that mounts the same claim:

kubectl run cache-inspect --rm -i --restart=Never \
  --image=public.ecr.aws/docker/library/busybox:1.36 \
  --overrides='{"spec":{"volumes":[{"name":"cache","persistentVolumeClaim":{"claimName":"binaries-cache"}}],"containers":[{"name":"cache-inspect","image":"public.ecr.aws/docker/library/busybox:1.36","command":["ls","-laR","/cache"],"volumeMounts":[{"name":"cache","mountPath":"/cache"}]}]}}'

After the first run completes, the binaries cache contains one directory for each tool, for example tofu/tofu-1.12.1, and a worker/ directory for the Spacelift worker binary. The provider cache contains one directory for each registry, such as registry.opentofu.org/hashicorp/aws/6.0.0/linux_amd64/. UID 1983 owns all of these files. Later runs reuse the files instead of downloading them again, including runs on other nodes.

Operational notes

• The NFS server runs as a single pod. If Kubernetes reschedules it, for example during a node replacement, the EBS volume is reattached to the new node. Mount operations in run pods block until the server returns because NFS uses the hard mount option by default. This usually takes less than a minute. Runs are delayed, but their data is not corrupted. The server pod can only run in the availability zone where the EBS volume was created.
• Volumes provisioned by the NFS server are world-writable, so no fsGroup is needed for UID 1983. For other volume types you may need spec.pod.securityContext.fsGroup: 1983 or adjusted export permissions.
• Cache contents do not contain source code or run state, but provider plugins are executable. Follow the cache security guidance, and do not use this StorageClass for the workspace volume.