With Nickel releasing 1.0 I’m excited to announce the 0.1 release of Tf-Ncl, an experimental tool for writing Terraform deployments with Nickel instead of HCL.

Tf-Ncl enables configurations to be checked against Terraform provider-specific contracts, before calling Terraform to perform the deployment. Nickel can natively generate outputs as JSON, YAML or TOML; since Terraform can accept its deployment configuration as JSON, you can straightforwardly export a Nickel configuration, adhering to the right format, to Terraform. Tf-Ncl provides a framework for ensuring a Nickel configuration has this specific format. Specifically, Tf-Ncl is a tool to generate Nickel contracts that describe the configuration schema expected by a set of Terraform providers.

This approach means that Terraform doesn’t need to know or care that Nickel has generated its deployment configuration. State management is entirely unaffected. And deployments written with Nickel can instruct Terraform to use existing HCL modules, making it possible to migrate a configuration incrementally. You can start using Nickel’s programming features without committing to a complete rewrite of all your configuration at once. Having the full power of Nickel available makes it possible to describe the important parameters of your deployment in a format that suits your application while minimizing duplication. Then you can write Nickel code to generate the necessary Terraform resource definitions in all their complexity. For example, you could maintain a list of user accounts with associated data like team membership and admin status, and then generate appropriate Terraform resources setting up the referenced teams and their member accounts. Later in this post, I’ll show you how to achieve a simplified version of this.

Tf-Ncl is a tech demo to show what is possible with Nickel and should be considered experimental at this time. But we do hope to improve it and your feedback will be essential for that.

Trying It Out

The quickest and easiest way to set up an example project is to use Nix flakes:

nix flake init -t github:tweag/tf-ncl#hello-tf

This will leave you with two files in the current directory, flake.nix and main.ncl. The flake.nix file defines a Nix flake which provides a shell environment with: nickel, the Nickel CLI; nls, the Nickel language server; and topiary, Tweag’s Tree-sitter based formatter. It also contains shell scripts to link the generated Nickel contracts into the current directory and to call Terraform with the result of a Nickel evaluation. Enter the development shell environment with:

nix develop

Now you can evaluate the Nickel configuration in main.ncl using:

run-nickel

Calling run-nickel doesn’t perform any Terraform operations yet, it just evaluates the Nickel code in main.ncl to produce a JSON file main.tf.json. The latter can be understood by Terraform and is treated just like an HCL configuration would be. In the hello-tf example, the deployment consists of a single null_resource with a local-exec provisioner that just prints Hello, world!. Continuing with our example, you can now initialize Terraform and apply the Terraform deployment to get your greeting:

terraform init
terraform apply

You can also combine the Nickel evaluation with the call to Terraform using the run-terraform wrapper script:

run-terraform apply

Let’s take a look at this tiny example deployment. It is configured in main.ncl:

let Tf = import "./tf-ncl-schema.ncl" in
{
  config.resource.null_resource.hello-world = {
    provisioner.local-exec = [
      { command = "echo 'Hello, world!'" }
    ],
  },
} | Tf.Config

This Nickel code first imports the contracts generated by Tf-Ncl and binds them to the name Tf. Then it defines a record which contains the overall configuration and declares it to be a Terraform configuration using the syntax | Tf.Config. For this toy example the deployment consists of just a null_resource with an attached local provisioner, that greets everyone it sees.

Let’s try to use this scaffolding for writing an example deployment. Let’s say we want to take a list of GitHub user names and add those to our GitHub organization.

The first thing to do is to declare to Tf-Ncl that we want to use the github Terraform provider. This can be done by adjusting the flake.nix file. The outputs section of the flake defines a devShell using a Tf-Ncl provided function. This function is what we need to customize:

outputs = inputs: inputs.utils.lib.eachDefaultSystem (system:
  {
    devShell = inputs.tf-ncl.lib.${system}.mkDevShell {
      providers = p: {
        inherit (p) null;
      };
    };
  });

This is the place were you can specify which Terraform providers your deployment will need. These are also the providers for which Tf-Ncl will generate Nickel contracts. To have Tf-Ncl generate contracts for the GitHub Terraform provider as well as the Terraform internal null provider, you would replace the function passed as providers to the mkDevShell function, i.e.:

providers = p: {
  inherit (p) null github;
};

Having done that, you need to re-enter the development environment by exiting the current one and running nix develop again. Afterwards the wrapper scripts run-nickel and run-terraform will all use the new contracts including the GitHub provider. Now, let’s write some Nickel to turn a list of GitHub user names into Terraform resources. Start with the hello-tf scaffold, remove the null_resource and add the users list:

let Tf = import "./tf-ncl-schema.ncl" in
{
  users = [ "alice", "bob", "charlie" ],
  config = {
    provider.github = [
      {
        token = "<placeholder-token>", # Don't do this in production!
        owner = "<placeholder-organization>",
      }
    ],
  }
} | Tf.Config

I’ve also added a provider section that will tell the GitHub Terraform provider which organization it should manage. If you do this for real, don’t put an authorization token in the configuration directly. Rather, use Terraform variables or data sources to retrieve secrets. The next step will be to process the list of usernames into github_membership resource blocks for Terraform. For that, you can use Nickel’s standard library to map over the users array.

This will leave you with an array of records. But what’s needed is a single record containing all the fields. The Nickel library function std.record.merge_all provides that functionality. Nickel has the F# and OCaml inspired |> operator which makes writing these kinds of pipelined function application quite ergonomic. Here’s how to use it for defining memberships:

memberships =
  users
  |> std.array.map (fun user => {
    resource.github_membership."%{user}-membership" = {
      username = user,
      role = "member",
    },
  })
  |> std.record.merge_all

Finally, the resulting memberships record needs to be combined with the provider configuration in the field config. That can be done with Nickel’s merging operator &. In summary, here’s the deployment:

let Tf = import "./tf-ncl-schema.ncl" in
{
  users = [ "alice", "bob", "charlie" ],
  memberships = users
    |> std.array.map
      (fun user => {
        resource.github_membership."%{user}-membership" = {
          username = user,
          role = "member",
        }
      })
    |> std.record.merge_all,
  config = {
    provider.github = [{
      token = "<placeholder-token>",
      owner = "<placeholder-organization>",
    }],
  }
  & memberships,
} | Tf.Config

Try to have Terraform generate a plan for the deployment:

$ run-terraform plan

Terraform will perform the following actions:

  # github_membership.alice-membership will be created
  + resource "github_membership" "alice-membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + username = "alice"
    }

  # github_membership.bob-membership will be created
  + resource "github_membership" "bob-membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + username = "bob"
    }

  # github_membership.charlie-membership will be created
  + resource "github_membership" "charlie-membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + username = "charlie"
    }

Plan: 3 to add, 0 to change, 0 to destroy.

It works 🎉 You can take a look at the entire example in the Tf-Ncl repository or by using a Nix flake template:

$ nix flake init -t github:tweag/tf-ncl#github-simple

If you happen to have existing HCL modules, those can be included in the Nickel configuration for an incremental migration. For example, let’s say example-module/main.tf contains the following module:

variable "greeting" {
  type = string
}

resource "null_resource" "greeter" {
  provisioner local-exec {
      command = "echo ${var.greeting}"
  }
}

Then this can be included from the top-level main.ncl by modifying the config attribute to include an instruction to Terraform to instantiate the module with some parameters. That is, you could use the following:

{
  # [...]
  config = {
    # [...]
    module.greeter = {
      source = "./example-module",
      greeting = "Hello, world!",
    }
  }
  & memberships,
} | Tf.Config

Future Directions

At this point, Tf-Ncl should be considered a tech demo for Nickel. While it can produce working deployments for Terraform, there are various areas that still need improvement. For one, the generated contracts can be huge for featureful providers. While this is actually a great benchmark for Nickel’s evaluator, it can cause problems; for example, asking the Nickel language server for completion candidates may time out for very large contracts. I’m looking into changing the structure of the Tf-Ncl contracts to make them more modular and easier to process piecewise. There are also limitations to Tf-Ncl’s handling of provider computed fields for Terraform. But more on that in a coming deep dive blog post on the technical challenges of building Tf-Ncl.

Tf-Ncl is a new tool. Feedback is essential for improving it. Please try out Nickel and Tf-Ncl, find new uses, break it and, most importantly, tell us about it!