Caching for Docker-based workers

Docker-based workers cache tool binaries automatically on each host. See Caching for an explanation of tool and provider caches, storage requirements and cache limitations.

This page explains how to share the Terraform/OpenTofu provider cache between workers.

Configure the provider cache

The configuration has two parts:

1. A directory on shared storage, mounted on every worker instance and bind-mounted into run containers with SPACELIFT_WORKER_EXTRA_MOUNTS.
2. TF_PLUGIN_CACHE_DIR set inside the run container, pointing at the mount.

The storage must meet the shared storage requirements. On AWS, you can use Amazon FSx for OpenZFS or run an NFS server backed by an EBS volume.

Example: NFS server backed by EBS

This example uses one dedicated EC2 instance as the NFS server. A gp3 EBS volume is attached to that instance and mounted at /srv/spacelift-provider-cache. Worker instances mount the exported directory over NFS.

The EBS volume is attached only to the NFS server. The workers do not attach the volume, and EBS Multi-Attach is not used.

Plan for NFS server resilience

This example is a starting point and does not configure high availability or failover for the NFS server. If the server is unavailable, the NFS hard mount option causes runs that access the cache to block until the server returns.

For production use, design the NFS server for the level of resilience your organization requires. The EBS volume must remain in its availability zone, and replacement servers need the same network endpoint or require workers to remount the export. How to implement that resilience is outside the scope of this guide.

If you do not want to manage this recovery process, consider using Kubernetes workers. The Kubernetes caching guide shows how to run the NFS server as a StatefulSet so that Kubernetes can reschedule it and reattach the EBS volume.

Configure the NFS server

The following Terraform creates a dedicated NFS server, an encrypted gp3 EBS volume and a security group that allows NFS traffic from the workers. The NFS setup runs through the server's cloud-init user data.

resource "aws_ebs_volume" "provider_cache" {
  availability_zone = var.nfs_server_availability_zone
  encrypted         = true
  size              = 50
  type              = "gp3"

  tags = {
    Name = "spacelift-provider-cache"
  }
}

resource "aws_security_group" "provider_cache_nfs" {
  name   = "spacelift-provider-cache-nfs"
  vpc_id = var.vpc_id

  ingress {
    from_port       = 2049
    to_port         = 2049
    protocol        = "tcp"
    security_groups = var.worker_pool_security_groups
  }
}

resource "aws_instance" "provider_cache_nfs" {
  ami                    = var.nfs_server_ami_id
  instance_type          = "t3.small"
  subnet_id              = var.nfs_server_subnet_id
  vpc_security_group_ids = [aws_security_group.provider_cache_nfs.id]

  user_data = <<-EOF
    #!/bin/bash
    set -euo pipefail

    dnf install -y nfs-utils

    EBS_DEVICE="/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_${replace(aws_ebs_volume.provider_cache.id, "-", "")}"
    until [ -b "$${EBS_DEVICE}" ]; do sleep 2; done

    if ! blkid "$${EBS_DEVICE}" >/dev/null 2>&1; then
      mkfs.xfs "$${EBS_DEVICE}"
    fi

    mkdir -p /srv/spacelift-provider-cache
    mount "$${EBS_DEVICE}" /srv/spacelift-provider-cache
    chown 1983:1983 /srv/spacelift-provider-cache
    chmod 0750 /srv/spacelift-provider-cache

    UUID=$(blkid -s UUID -o value "$${EBS_DEVICE}")
    echo "UUID=$${UUID} /srv/spacelift-provider-cache xfs defaults,nofail 0 2" >> /etc/fstab

    echo "/srv/spacelift-provider-cache ${var.worker_subnet_cidr}(rw,sync,no_subtree_check,root_squash)" \
      > /etc/exports.d/spacelift-provider-cache.exports

    systemctl enable --now nfs-server
    exportfs -rav
  EOF

  tags = {
    Name = "spacelift-provider-cache-nfs"
  }
}

resource "aws_volume_attachment" "provider_cache" {
  device_name = "/dev/sdf"
  instance_id = aws_instance.provider_cache_nfs.id
  volume_id   = aws_ebs_volume.provider_cache.id
}

The filesystem is formatted only when the volume is empty, so replacing the NFS server does not erase an existing cache. Run containers execute as UID 1983, which owns the cache directory. Mode 0750 prevents other users on the NFS server from reading or writing it.

The example limits NFS access to the worker security groups and subnet CIDR, and uses root_squash to prevent client root users from acting as root on the export. See Cache security before sharing a cache between stacks with different trust levels.

Configure the workers

The steps below assume that you deployed the pool with the terraform-aws-spacelift-workerpool-on-ec2 module and the Spacelift worker AMI. The AMI uses Amazon Linux 2023 minimal, which does not include an NFS client.

Mount the export during instance startup and expose it to run containers. The module adds its configuration variable to each worker's cloud-init user data before the launcher starts. The launcher inherits any variables exported there:

module "my_workerpool" {
  source = "github.com/spacelift-io/terraform-aws-spacelift-workerpool-on-ec2"

  configuration = <<EOF
    dnf install -y nfs-utils
    mkdir -p /mnt/spacelift-provider-cache
    until mount -t nfs4 -o nfsvers=4.1,hard,timeo=600,retrans=2 \
      ${aws_instance.provider_cache_nfs.private_dns}:/srv/spacelift-provider-cache \
      /mnt/spacelift-provider-cache; do
      sleep 5
    done

    export SPACELIFT_WORKER_EXTRA_MOUNTS="/mnt/spacelift-provider-cache:/mnt/provider-cache"
    export TF_PLUGIN_CACHE_DIR=/mnt/provider-cache
    export SPACELIFT_WHITELIST_ENVS=TF_PLUGIN_CACHE_DIR
  EOF

  # ... the rest of your pool configuration

  depends_on = [aws_volume_attachment.provider_cache]
}

• SPACELIFT_WORKER_EXTRA_MOUNTS bind-mounts the host directory into every run container at /mnt/provider-cache.
• SPACELIFT_WHITELIST_ENVS passes TF_PLUGIN_CACHE_DIR from the launcher's environment into every run. This enables the cache for all stacks on the pool. To enable it only for selected stacks, remove the last two export lines and set TF_PLUGIN_CACHE_DIR=/mnt/provider-cache on those stacks instead.

If you set TF_PLUGIN_CACHE_MAY_BREAK_DEPENDENCY_LOCK_FILE=true as described in Dependency lock files, also add it to SPACELIFT_WHITELIST_ENVS.

If you use the CloudFormation template, put the same commands in the secret referenced by CustomUserDataSecretName. See Injecting custom commands during instance startup.

Verify the cache

After a run completes, the mount on each worker instance contains a directory for each registry, such as registry.opentofu.org/hashicorp/aws/6.0.0/linux_amd64/. On later runs, init reports that it is using providers from the cache instead of downloading them.