Migrating to Spacelift»
Migrating from one Infrastructure as Code CI/CD provider to another can feel daunting. This is why we created a migration kit that takes care of the heavy lifting.
Edit a few settings, and let it do the hard work. Then review, and possibly tweak, the generated code, and finally have your Spacelift entities created.
There is no one-size-fits-all for this kind of migration. This is why we designed this tool to be flexible and easy to hack to meet your specific needs. Feel free to reach out to our support team if you need any help or guidance.
Overview»
The migration process is as follows:
- Export the definition for your resources at your current vendor.
- Generate the Terraform code to recreate similar resources at Spacelift using the Terraform provider.
- Review and possibly edit the generated Terraform code.
- Commit the Terraform code to a repository.
- Create a manager Spacelift stack that points to the repository with the Terraform code.
Tip
Currently, only Terraform Cloud and Terraform Enterprise are supported as sources. The instructions below apply to both.
Prerequisites»
Installation»
- Ensure that Python is installed.
- Download the Migration Kit:
git clone git@github.com:spacelift-io/spacelift-migration-kit.git
(or other available methods in GitHub). - Go to the Migration Kit folder:
spacelift-migration-kit
. - Install the Python dependencies and the
spacemk
command in a Python virtual environment:poetry install
. - Activate the Python virtual environment:
poetry shell
.
Usage»
Configuration»
Copy the config.yml.example
file to config.yml
and edit it as needed.
Environment variables can be referenced by their name preceded by the $
sign (e.g., $API_TOKEN
).
This is helpful if you do not want to store sensitive information in the configuration file.
If a .env
file is present at the root of the Spacelift Migration Kit folder, it will be automatically loaded when running spacemk
and the tests, and the environment variables it contains will be available to that process.
Audit»
This step is optional but recommended. It will analyze your current setup and display statistics in the terminal. Also, an Excel file with the list of entities to be migrated is created (tmp/report.xlsx
).
Additionally, it will perform checks and warn you of possible problems. For example, entities cannot be automatically migrated and might need to be handled manually.
Migration»
The migration is split into a few different steps that need to be run in order.
Export»
The spacemk export
command exports information about the source provider entities and stores them as a normalized JSON file (tmp/data.json
).
That file can be reviewed and modified before moving to the next step.
Generate»
The spacemk generate
command uses the normalized JSON file from the export step and uses a Jinja template to generate Terraform code that uses the Spacelift provider to create Spacelift entities that mimic the behavior of the source provider entities.
The generated code can be found in the tmp/code/main.tf
file. Feel free to review and edit it as needed.
Publish»
Once the Terraform code has been generated, push the tmp/code/main.tf
file to a git repository of your choosing that is available to your Spacelift account.
Deploy»
After pushing the generated Terraform has been pushed to a git repository, create a manager stack in Spacelift.
Point it to the repository, and possibly folder, where you stored the Terraform code, and make sure to mark it as administrative.
Finally, trigger a run to create the Spacelift entities.
Set Sensitive Variable Values»
This step can be skipped if there are no sensitive variables defined.
To avoid storing sensitive variable values in Terraform code and the state file, the generate
command does not set the value for those variables.
Once the stacks have been created, set the values for the spacelift
section of the config.yml
file and run the
spacemk set-sensitive-env-vars
command to set the value for the sensitive environment variables.
Set Terraform Variables with Invalid Names»
This step can be skipped if there are no Terraform variables with an invalid name.
Among the different ways to pass variable values to Terraform, Spacelift uses environment variables named TF_VAR_
followed by the name of a declared variable.
However, Terraform allows the use of characters in variable names that are not allowed in environment variable names (e.g., -
).
To work around this issue, the Spacelift Migration Kit identifies Terraform variables with invalid names and stores them in a mounted file named tf_vars_with_invalid_name.auto.tfvars
so that it gets automatically loaded by Terraform.
Once the stacks have been created, set the values for the spacelift
section of the config.yml
file and run the
spacemk set-tf-vars-with-invalid-name
command to set the values for the Terraform variables with invalid names.
Create Module Versions»
This step can be skipped if there are no modules defined.
Once the modules have been created, set the values for the github
section of the config.yml
file and run the
spacemk create-module-versions
command to re-create existing module versions.
Cleanup»
All temporary local artifacts are stored in the tmp
folder. Delete some or all of it to clean up.
Additionally, you can destroy the Spacelift resources created by the manager stack and then the manager stack to fully remove the migration artifacts.
The source vendor setup is left untouched by the Migration Kit and can be deleted once the migration has been verified to be successful.
Customization»
Every migration is different, and while the Spacelift Migration Kit aims at doing most of the heavy lifting, there is often a need for customizing the workflow.
Spacelift Migration Kit has been designed to be easily extended and modified. All customizations are stored in the custom
folder.
Custom Template»
The generate
command uses a Jinja template that can be overridden partially or entirely by creating a file named main.tf.jinja
in the custom/templates
folder.
To selectively override pieces of the base template, add the following instruction at the top of the custom template:
1 |
|
Then, override any block by declaring it in the custom template.
Here is an example:
1 2 3 4 5 6 7 |
|
Available blocks can be found in the base.tf.jinja template.
It should be rarely needed, but if you can override the base template entirely by not including the {% extends "base.tf.jinja" %}
instruction.
Custom Command»
You can add a custom command by creating a Python file in the custom/commands
folder based on the following code:
1 2 3 4 5 |
|
The file can have any name, but we recommend naming it after the command name. In the example above, the file would be custom.py
.
If the custom command needs some configuration settings, they can be added to the config.yml
file, and the configuration passed to the command:
1 2 3 4 5 6 |
|
The commands are managed by the click Python library. Check its documentation or the Spacelift Migration Kit native commands for examples.
Custom Exporter»
There might be no native exporter for your source provider, or you might need to tweak an existing provider.
To do so, you can create a Python file in the custom/exporters
folder. It must be named after the exporter and define a class named <CapitalizedExporterName>Exporter
that derives from the spacemk.exporters.BaseExporter
class for new exporters or a native exporter class when overriding an existing exporter.
Here are a few examples:
Exporter Name | Filename | Class Name |
---|---|---|
foo |
foo.py |
FooExporter |
foo_bar |
foo_bar.py |
FooBarExporter |
Here is an example:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
|
Custom Python Packages»
If the custom Python code requires packages not included in Spacelift Migration Kit, you can create a requirements.txt
file in the custom
folder and install those dependencies with the following command:
1 |
|
Storing Customizations»
Simple Use Case»
Customizations live in the custom
folder. This is fine for most use cases but could be a problem if more than one engineer works on the migration or if you need to collaborate with Spacelift engineers on advanced migrations.
Advanced Use Case»
For those advanced use cases, the proposed approach is to create a private clone of this repository and version your customizations there.
Here are the steps to create the private clone:
- Create a bare clone of the repository. This is temporary and will be removed, so just do it wherever.
1 |
|
-
Create a new private repository in your VCS provider and name it
spacelift-migration-kit
. -
Mirror-push your bare clone to your new
spacelift-migration-kit
repository.
1 2 |
|
- Remove the temporary local clone you created in step 1.
1 2 |
|
- You can now clone your
spacelift-migration-kit
repository on your machine where you see fit.
1 |
|
- Add the original
spacelift-migration-kit
repository asupstream
to fetch updates. The example below uses GitHub but you can use any git VCS provider information.
1 2 |
|
- The git remotes for your local clone, listed with
git remote -v
should look like this:
1 2 3 4 |
|
- Interact with your
origin
remote as usual. You can pull changes from the original repository by fetching from theupstream
remote and rebasing on top of your local branch.
1 2 |
|
There should not be any conflicts if you keep your modifications in the custom
folder, but if there are, solve them as usual.
Uninstallation»
- Delete the Python virtual environment:
poetry env remove --all
. - Delete the
spacelift-migration-kit
folder.
Support»
If you found a bug or want to submit a feature request, please use the repository issues.
If you need help or guidance, please reach out to your Solutions Engineer or our support.