Drift detection

Drift happens

In infrastructure-as-code, the concept of drift represents the difference between the desired and the actual state of the infrastructure managed by your tool of choice - Terraform, Pulumi, AWS CloudFormation, etc. In practice, there are two sources of drift.

The first source covers changes directly introduced by external actors - either humans or machines (scripts). If an on-call SRE changes your database parameters otherwise controlled by Terraform, you've introduced drift. If an external script updates your Kubernetes cluster in a way that conflicts with its Pulumi definition, it's drift as well.

The other source of drift comes from the dependency of your resources on external data sources. For example, if your load balancer only expects to receive traffic from Cloudflare, you may want to restrict ingress to a predefined range of IPs. However, that range may be dynamic, and your IaC tool queries it every time it runs. If there's any change to the external data source, it's showing up as drift, too.

In the first scenario, drift is an unwanted by-product of emergencies or broken processes. In the latter, it's both desired and inevitable - it's proof that your otherwise declarative system responds to external changes. In other words - drift happens, so deal with it. 😎

Video Walkthrough

How Spacelift helps

Spacelift comes with a built-in mechanism to detect and - optionally - reconcile drift. We do it by periodically executing proposed runs on your stable infrastructure (in Spacelift, we generally represent it by the FINISHED stack state) and checking for any changes.

To get started, create a drift detection configuration from the Scheduling section of your stack. You will be able to add multiple cron rules to define when your reconciliation jobs should be scheduled, as well as decide whether you want your jobs to trigger tracked runs (reconciliation jobs) in response to detected drift:

Info

Note that, at least currently, drift detection only works on private workers.

To reconcile, or not to reconcile

We generally suggest turning reconciliation "on" as it ensures that you get the most out of drift detection. Reconciliation jobs are equivalent to manually triggering tracked runs and obey the same rules and constraints. In particular, they respect their stacks' auto-deploy setting and trigger plan policies - see this section for more details.

However, if you choose not to reconcile changes, you can still get value out of drift detection - in this case, drifted resources can be seen in the Resources view, both on the stack and account level. Also, drift detection jobs trigger webhooks like regular runs, where they're clearly marked as such (driftDetection field).

Drift detection in practice

With drift detection enabled on the stack, proposed runs are quietly executing in the background. If they do not detect any changes, the only way you'd know about them is by viewing all runs in the Account > Runs section and filtering or grouping by drift detection parameter - here is an example:

But once your job detects drift (and you've enabled reconciliation), it triggers a regular tracked run. This run is subject to the same rules as a regular tracked run is. For example, if you set your stack not to deploy changes automatically, the run will end up in an Unconfirmed state, waiting for your decision. The same thing will happen if a plan policy produces a warning using a matched warn rule.

Policy input

The only real difference between a drift detection job and one triggered manually is that the run section of your policy input will have the drift_detection field set to true - and this applies to both plan and trigger policies. You can use this mechanism to add extra controls to your drift detection strategy. For example, if you're automatically deploying your changes but want a human to look at drift before reconciling it, you can add the following section to your plan policy body:

package spacelift

warn["Drift reconciliation requires manual approval"] {
  input.spacelift.run.drift_detection
}