However, if you ask a random developer, I can guarantee that one aspect will be mentioned every time: algebraic data types (ADTs) and pattern matching. They are the bread and butter of typed functional languages. ADTs are relatively easy to implement (for language maintainers) and easy to use. They’re part of the 20% of the complexity that makes for 80% of the joy of functional programming.

But Nickel didn’t have ADTs until recently. In this post, I’ll tell the story of Nickel and ADTs, starting from why they were initially lacking, the exploration of different possible solutions and the final design leading to the eventual retro-fitting of proper ADTs in Nickel. This post is intended for Nickel users, for people interested in configuration management, but also for anyone interested in programming language design and functional programming. It doesn’t require prior Nickel knowledge.

A quick primer on Nickel

Nickel is a gradually typed, functional, configuration language. From this point, we’ll talk about Nickel before the introduction of ADTs in the 1.5 release, unless stated otherwise. The core language features:

• let-bindings: let extension = ".ncl" in "file.%{extension}"
• first-class functions: let add = fun x y => x + y in add 1 2
• records (JSON objects): {name = "Alice", age = 42}
• static typing: let mult : Number -> Number -> Number = fun x y => x * y. By default, expressions are dynamically typed. A static type annotation makes a definition or an inline expression typechecked statically.
• contracts look and act almost like types but are evaluated at runtime: { port | Port = 80 }. They are used to validate configurations against potentially complex schemas.

The lifecycle of a Nickel configuration is to be 1) written, 2) evaluated and 3) serialized, typically to JSON, YAML or TOML. An important guideline that we set first was that every native data structure (record, array, enum, etc.) should be trivially and straightforwardly serializable to JSON. In consequence, Nickel started with the JSON data model: records (objects), arrays, booleans, numbers and strings.

There’s one last primitive value: enums. As in C or in JavaScript, an enum in Nickel is just a tag. An enum value is an identifier with a leading ', such as in {protocol = 'http, server = "tweag.io"}. An enum is serialized as a string: the previous expression is exported to JSON as {"protocol": "http", "server": "tweag.io"}.

So why not just using strings? Because enums can better represent a finite set of alternatives. For example, the enum type [| 'http, 'ftp, 'sftp |] is the type of values that are either 'http, 'ftp or 'sftp. Writing protocol : [| 'http, 'ftp, 'sftp |] will statically (at typechecking time) ensure that protocol doesn’t take forbidden values such as 'https. Even without static typing, using an enum conveys to the reader that a field isn’t a free-form string.

Nickel has a match which corresponds to C or JavaScript’s switch:

is_http : [| 'http, 'ftp, 'sftp |] -> Bool =
  match {
    'http => true,
    _ => false,
  }

As you might notice, there are no ADTs in sight yet.

ADTs in a configuration language

While Nickel is a functional language, it’s first and foremost a configuration language, which comes with specific design constraints.

Because we’re telling the story of ADTs before they landed in Nickel, we can’t really use a proper Nickel syntax yet to provide examples. In what follows, we’ll use a Rust-like syntax to illustrate the examples: enum Foo<T> { Bar(i32), Baz(bool, T) } is an ADT parametrized by a generic type T with two constructors Bar and Baz, where the first one takes an integer as an argument and the other takes a pair of a boolean and a T. Concrete values are written as Bar(42) or Baz(true, "hello").

An unexpected obstacle: serialization

As said earlier, we want values to be straightforwardly serializable to the JSON data model.

Now, take a simple ADT such as enum Foo<T,U> = { SomePair(T,U), Nothing }. You can find reasonable serializations for SomePair(1,2), such as {"tag": "SomePair", "a": 1, "b": 2}. But why not {"flag": "SomePair", "0": 1, "1": 2} or {"mark": "SomePair", "data": [1, 2]}? While those representations are isomorphic, it’s hard to know the right choice for the right use-case beforehand, as it depends on the consumer of the resulting JSON. We really don’t want to make an arbitrary choice on behalf of the user.

Additionally, while ADTs are natural for a classical typed functional language, they might not entirely fit the configuration space. A datatype like enum Literal { String(String), Number(Number) } that can store either a string or a number is usually represented directly as an untagged union in a configuration, that {"literal": 5} or {"literal": "hello"}, instead of the less natural tagged union (another name for ADTs) {"literal": {"tag": "Number", "value": 5}}.

This led us to look at (untagged) union types instead. Untagged unions have the advantage of not making any choice about the serialization: they aren’t a new data structure, as are ADTs, but rather new types (and contracts) to classify values that are already representable.

The road of union types

A union type is a type that accepts different alternatives. We’ll use the fictitious \/ type combinator to write a union in Nickel (| is commonly used elswhere but it’s already taken in Nickel). Our previous example of a literal that can be either a string or a number would be {literal: Number \/ String}. Those types are broadly useful independently of ADTs. For example, JSON Schema features unions through the core combinator any_of.

Our hope was to kill two birds with one stone by adding unions both as a way to better represent existing configuration schemas, but also as a way to emulate ADTs. Using unions lets users represent ADTs directly as plain records using their preferred serialization scheme. Together with flow-sensitive typing, we can get as expressive as ADTs while letting the user decide on the encoding. Here is an example in a hypothetical Nickel enhanced with unions and flow-sensitive typing:

let sum
  : {tag = 'SomePair, a : Number, b : Number} \/ {tag = 'Nothing}
    -> Number
  = match {
    {tag = 'SomePair, a, b} => a + b,
    {tag = 'Nothing} => 0,
  }

Using unions and flow-sensitive typing as ADTs is the approach taken by TypeScript, where the previous example would be:

type Foo = { tag: "SomePair"; a: number; b: number } | { tag: "Nothing" }

function sum(value: Foo): number {
  switch (value.tag) {
    case "SomePair":
      return value.a + value.b
    case "Nothing":
      return 0
  }
}

In Nickel, any type must have a contract counter-part. Alas union and intersection contracts are hard (in fact, union types alone are also not a trivial feat to implement!). In the linked blog post, we hint at possible pragmatic solutions for union contracts that we finally got to implement for Nickel 1.8. While sufficient for practical union contracts, this is far from the general union types that could subsume ADTs. This puts a serious stop to the idea of using union types to represent ADTs.

What are ADTs really good for?

As we have been writing more and more Nickel, we realized that we have been missing ADTs a lot for library functions - typically the types enum Option<T> { Some(T), None } and Result<T,E> = { Ok(T), Error(E) } - where we don’t care about serialization. Those ADTs are “internal” markers that wouldn’t leak out to the final exported configuration.

Here are a few motivating use-cases.

std.string.find

std.string.find is a function that searches for a substring in a string. Its current type is:

String
-> String
-> { matched : String, index : Number, groups : Array String }

If the substring isn’t found, {matched = "", index = -1, groups []} is returned, which is error-prone if the consumer doesn’t defend against such values. We would like to return a proper ADT instead, such as Found {matched : String, index : Number, groups : Array String} or NotFound, which would make for a better and a safer interface¹.

Contract definition

Contracts are a powerful validation system in Nickel. The ability to plug in your own custom contracts is crucial.

However, the general interface to define custom contracts can seem bizarre. Custom contracts need to set error reporting data on a special label value and use the exception-throwing-like std.contract.blame function. Here is a simplified definition of std.number.Nat which checks that a value is natural number:

fun label value =>
  if std.typeof value == 'Number then
    if value % 1 == 0 && value >= 0 then
      value
    else
      let label = std.contract.label.with_message "not a natural" in
      std.contract.blame label
  else
    let label = std.contract.label.with_message "not a number" in
    std.contract.blame label

There are good (and bad) reasons for this situation, but if we had ADTs, we could cover most cases with an alternative interface where custom contracts return a Result<T,E>, which is simpler and more natural:

fun value =>
  if std.typeof value == 'Number then
    if value % 1 == 0 && value >= 0 then
      Ok
    else
      Error("not a natural")
  else
    Error("not a number")

Of course, we could just encode this using a record, but it’s just not as nice.

Let it go, let it go!

The list of other examples of using ADTs to make libraries nicer is endless.

Thus, for the first time, we decided to introduce a native data structure that isn’t serializable.

Note that this doesn’t break any existing code and is forward-compatible with making ADTs serializable in the future, should we change our mind and settle on one particular encoding. Besides, another feature is independently explored to make serialization more customizable through metadata, which would let users use custom (de)serializer for ADTs easily.

Ok, let’s add the good old-fashioned ADTs to Nickel!

The design

Structural vs nominal

In fact, we won’t exactly add the old-fashioned version. ADTs are traditionally implemented in their nominal form.

A nominal type system (such as C, Rust, Haskell, Java, etc.) decides if two types are equal based on their name and definition. For example, values of enum Alias1 { Value(String) } and enum Alias2 { Value(String) } are entirely interchangeable in practice, but Rust still doesn’t accept Alias1::Value(s) where a Alias2 is expected, because those types have distinct definitions. Similarly, you can’t swap a class for another in Java just because they have exactly the same fields and methods.

A structural type system, on the other hand, only cares about the shape of data. TypeScript has a structural type system. For example, the types interface Ball { diameter: number; } and interface Sphere { diameter: number; } are entirely interchangeable, and {diameter: 42} is both a Ball and a Sphere. Some languages, like OCaml² or Go³, mix both.

Nickel’s current type system is structural because it’s better equipped to handle arbitrary JSON-like data. Because ADTs aren’t serializable, this consideration doesn’t weight as much for our motivating use-cases, meaning ADTs could still be either nominal or structural.

However, nominal types aren’t really usable without some way of exporting and importing type definitions, which Nickel currently lacks. Structural ADTs look like the better choice for Nickel. We can build, typecheck, and match on ADTs locally without having to know or to care about any type declaration. Structural ADTs are a natural extension of Nickel (structural) enums, syntactically, semantically, and on the type level, as we will see.

While less common, structural ADTs do exist in the wild and they are pretty cool. OCaml has both nominal ADTs and structural ADTs, the latter being known as polymorphic variants. They are an especially powerful way to represent a non trivial hierarchy of data types with overlapping, such as abstract syntax trees or sets of error values.

Syntax

C-style enums are just a special case of ADTs, namely ADTs where constructors don’t have any argument. The dual conclusion is that ADTs are enums with arguments. We thus write the ADT Some("hello") as an enum with an argument in Nickel: 'Some "hello".

We apply the same treatment to types. [| 'Some, 'None |] was a valid enum type, and now [| 'Some String, 'None |] is also a valid type (which would correspond to Rust’s Option<String>).

There is a subtlety here: what should be the type inferred for 'Some now? In a structural type system, 'Some is just a free-standing symbol. The typechecker can’t know if it’s a constant that will stay as it is - and thus has the type [| 'Some |] - or a constructor that will be eventually applied, of type a -> [| 'Some a |]. This difficulty just doesn’t exist in a nominal type system like Rust: there, Option::Some refers to a unique, fixed ADT constructor that is known to require precisely one argument.

To make it work, 'Ok 42 isn’t actually a normal function application in Nickel: it’s an ADT constructor application, and it’s parsed differently. We just repurpose the function application syntax⁴ in this special case. 'Ok isn’t a function, and let x = 'Ok in x 42 is an error (applying something that isn’t a function).

You can still recover Rust-style constructors that can be applied by defining a function (eta-expanding, in the functional jargon): let ok = fun x => 'Ok x.

We restrict ADTs to a single argument. You can use a record to emulate multiple arguments: 'Point {x = 1, y = 2}.

ADTs also come with pattern matching. The basic switch that was match is now a powerful pattern matching construct, with support for ADTs but also arrays, records, constant, wildcards, or-patterns and guards (if side-conditions).

Typechecking

Typechecking structural ADTs is a bit different from nominal ADTs. Take the simple example (the enclosing : _ annotation is required to make the example statically typed in Nickel)

(
  let data = 'Ok 42 in
  let process = match {
    'Ok x => x + 1,
    'Error => 0,
  } in

  process data
) :  _

process is inferred to have type [| 'Ok Number, 'Error |] -> Number. What type should we infer for data = 'Ok 42? The most obvious one is [| 'Ok Number |]. But then [| 'Ok Number |] and [| 'Ok Number, 'Error |] don’t match and process data doesn’t typecheck! This is silly, because this example should be perfectly valid.

One possible solution is to introduce subtyping, which is able to express this kind of inclusion relation: here that [| 'Ok Number |] is included in [| 'Ok Number, 'Error |]. However, subtyping has some defects and is whole can of worms when mixed with polymorphism (which Nickel has).

Nickel rather relies on another approach called row polymorphism, which is the ability to abstract over not just a type, as in classical polymorphism, but a whole piece of an enum type. Row polymorphism is well studied in the literature, and is for example implemented in PureScript. Nickel already features row polymorphism for basic enum types and for records types.

Here is how it works:

let process : forall a. [| 'Ok Number, 'Error; a |] -> Number = match {
  'Ok x => x + 1,
  'Error => 0,
  _ => -1,
} in

process 'Other

Because there’s a catch-all case _ => -1, the type of process is polymorphic, expressing that it can handle any other variant beside 'Ok Number and 'Error (this isn’t entirely true: Ok String is forbidden for example, because it can’t be distinguished from Ok Number). Here, a can be substituted for a subsequence of an enum type, such as 'Foo Bool, 'Bar {x : Number}.

Equipped with row polymorphism, we can infer the type forall a. [| 'Ok Number; a |]⁵ for 'Ok 42. When typechecking process data in the original example, a will be instantiated to the single row 'Error and the example typechecks. You can learn more about structural ADTs and row polymorphism in the corresponding section of the Nickel user manual.

Conclusion

While ADTs are part of the basic package of functional languages, Nickel didn’t have them until relatively recently because of peculiarities of the design of a configuration language. After exploring the route of union types, which came to a dead-end, we settled on a structural version of ADTs that turns out to be a natural extension of the language and didn’t require too much new syntax or concepts.

ADTs already prove useful to write cleaner and more concise code, and to improve the interface of libraries, even in a gradually typed configuration language. Some concrete usages can be found in try_fold_left and validators already.

• Nickel merge is designed to combine several pieces of configurations which all respect the same contract. This allows an application developer to define the interface of its configuration, and have the user write it in a modular way.
• NixOS modules, on the other hand, are designed to combine pieces which not only define a part of the final configuration, but also a part of the contract. This is a must-have for big systems like NixOS, where defining the whole schema in one place isn’t possible.

Even in Nickel, it would be sometimes desirable to get the full modularity of NixOS modules. For instance, Organist is based on this idea of having individual pieces, each defining both a part of the final schema and a part of the configuration — the files module will define the interface for declaring custom config files, and hook into the base Nix system to declare a flake app based on it, the editorconfig module declares an interface for managing the .editorconfig file, and hooks into the files interface for the actual generation of the file, etc.

Fortunately, it is actually trivial to implement an equivalent of the NixOS module system in Nickel by leveraging the merge system. Not only that, but this will only be a very light abstraction over built-in features. This means that it will come with the joy of being understood by the LSP, giving nice auto-completion, quick and relevant error messages, and so on. Also, the lightness of the abstraction makes it very flexible, allowing to easily build variants of the system.

Before explaining how that works, let’s define a bit more precisely what we want with a module system.

Module systems

A “module system”, in the NixOS sense, is a programming paradigm which allows exploding a complex configuration into individual components. Each component (a “module”) can define both a piece of the schema for the overall configuration, and a piece of the configuration. The schemas of all the modules are combined to form the final schema, and the same goes for configurations. The only constraint is that the final configuration matches the final schema.

For instance, here is an instantiation of a (very) simplified version of the NixOS module system, written in Nix:

mergeModules {
   module1 = {...}: {
     options.foo = mkOption { type = int; };
     options.bar = mkOption { type = string; };
     config.bar = "world";
   };
   module2 = {config, ...}: {
      options.baz = mkOption { type = string; };
      config.foo = 3;
      config.baz = "Hello " + config.bar;
    };
 }

# Result
=> {
  foo = 3;              # defined by module1, set by module2
  bar = "world";        # defined by module1, set by module1
  baz = "Hello world";  # defined by module2, set by module2 using `bar` from module 1
}

We can see that each module defines both a piece of the final schema (the options field) and a piece of the final config (the config field). They have the ability to set and refer to options defined in another module.

Assigning a value of the wrong type to an option gives an error. For instance, changing config.foo = 3 to config.foo = "3" yields:

error:
       … while evaluating the attribute 'value'
         at /nix/store/axfhzzkixwwdmxlrma7k8f65214acvml-source/lib/modules.nix:809:9:
          808|     in warnDeprecation opt //
          809|       { value = builtins.addErrorContext "while evaluating the option `${showOption loc}':" value;
             |         ^
          810|         inherit (res.defsFinal') highestPrio;

       … while evaluating the option `foo':

       … while evaluating the attribute 'mergedValue'
         at /nix/store/axfhzzkixwwdmxlrma7k8f65214acvml-source/lib/modules.nix:844:5:
          843|     # Type-check the remaining definitions, and merge them. Or throw if no definitions.
          844|     mergedValue =
             |     ^
          845|       if isDefined then

       (stack trace truncated; use '--show-trace' to show the full, detailed trace)

       error: A definition for option `foo' is not of type `signed integer'. Definition values:
       - In `<unknown-file>': "foo"

Shortcomings of the NixOS module system

This system is incredibly powerful, and rightfully serves not only the whole of NixOS, but also a number of other projects in the Nix world, such as home-manager, nix-darwin, flake parts and terranix.

However, it suffers from some limitations, mostly due to it being a pure library-side encoding instead of a Nix built-in.

• The error messages can get quite daunting (just look at the example above). Great effort has been put to improve the Nix error messages (and it shows), but the situation is still far from ideal.
• Because it has nearly zero support from the language side, no LSP server is truly able to understand it, meaning that things like autocompletion or in-editor error messages are very much best-effort, if they exist at all.
• Finally, it is often too dynamic for its own good. The fact that it only cares about global consistency makes it nearly impossible to reason about a module individually. The only way, for instance, to know whether a given module is correct is to evaluate the whole module system to know whether all the options it refers to are defined somewhere. Likewise, it is absolutely impossible to know the exact consequences of flipping an option somewhere as it might have an impact on any other module.

Implementing a (better) module system in Nickel

Given the usefulness of this module system, and its weaknesses, it would be great if we could have the same thing in Nickel – and even better if we could fix the shortcomings on the way.

It turns out that we can, and in a whopping one line of code:

{ Module = { Schema | not_exported = {}, config | Schema } }

This probably needs some explanation, so let’s look at how it works, and how it can be used.

Before anything else, let’s be honest: the simplicity of that line is rather deceptive since it hides the big heavy lifting done by the runtime, in particular the merge and contract systems.

This defines a contract called Module that represents values with:

• A Schema field, which is a record contract;
• A config field, matching the contract defined by Schema.

We can use it as such:

let Module = { Schema | not_exported = {}, config | Schema } in

{
  module1 = {
    Schema.foo | Number,
    Schema.bar | String,
    config.bar = "world",
  },
  module2 = {
    Schema.baz | String,
    config.bar,
    config.foo = 3,
    config.baz = "Hello " ++ config.bar,
  },

  module3 | Module = module1 & module2,
}.module3.config

# Result
=> {
  "bar": "world",
  "baz": "Hello world",
  "foo": 3
}

This is very similar to the Nix example above. What happens if we make a mistake on one of the fields, say replace config.foo = 1 with config.foo = "x"?

$ nickel export main.ncl
error: contract broken by the value of `foo`
   ┌─ /tmp/tmp.U0m6Ro8Vvo/main.ncl:13:18
   │
 5 │     Schema.foo | Number,
   │                  ------ expected type
   ·
13 │     config.foo = "x",
   │                  ^^^ applied to this expression
   │
   ┌─ <unknown> (generated by evaluation):1:1
   │
 1 │ "x"
   │ --- evaluated to this value

So the contract system is aware of what we want to check, and reports us an error accordingly. Better, it points to the right place in the code. Even better, the LSP server is aware of the error, and can point right at it:

Global vs local consistency

This is definitely great, and solves the LSP integration issue, as well as part of the error messages one. What it doesn’t do is alleviate the lack of local consistency: module1 and module2 are still implicitly depending on each other, and the only way to know that is to see that — for instance — module2 sets bar, which is declared in module1.

We can, however, enforce something stricter: In our example, we only require module1 & module2 to be a valid module. But we could require each of them to be individually consistent by enforcing that their config field matches their Schema field:

--- a/main.ncl
+++ b/main.ncl
@@ -4,11 +4,13 @@ let Module = { Schema | not_exported = {}, config | Schema } in
   module1 = {
     Schema.foo | Number,
     Schema.bar | String,
+    config | Schema,
     config.bar = "world",
   },
   module2 = {
     Schema.baz | String,

+    config | Schema,
     config.bar,
     config.foo = 3,
     config.baz = "Hello " ++ config.bar,

Doing so will yield an error:

$ nickel export main.ncl
error: contract broken by the value of `config`
       extra fields `bar`, `foo`
   ┌─ /tmp/tmp.U0m6Ro8Vvo/main.ncl:13:14
   │
13 │     config | Schema,
   │              ------ expected type
   │
   ┌─ <unknown> (generated by evaluation):1:1
   │
 1 │ { bar, baz = %<closure@0x55fa68e718c8>, foo = 3, }
   │ -------------------------------------------------- evaluated to this value
   │
   = Have you misspelled a field?
   = The record contract might also be too strict. By default, record contracts exclude any field which is not listed.
     Append `, ..` at the end of the record contract, as in `{some_field | SomeContract, ..}`, to make it accept extra fields.

Indeed, module2 isn’t consistent. It is using foo and bar, but doesn’t know about them as they are declared in module1. But we can fix that by making it explicitly depend on module1:

--- a/main.ncl
+++ b/main.ncl
@@ -7,7 +7,7 @@ let Module = { Schema | not_exported = {}, config | Schema } in
     config | Schema,
     config.bar = "world",
   },
-  module2 = {
+  module2 = module1 & {
     Schema.baz | String,

     config | Schema,

And now everything gets completely consistent, and we can confidently write a fully modular configuration, with the assurance that the language will have our back and give us early warnings in case anything goes wrong.

Besides, since the language really knows about everything that’s going on, the LSP can do its magic and help us with autocompletion. Let’s add a new option to module1:

--- a/main.ncl
+++ b/main.ncl
@@ -4,6 +4,12 @@ let Module = { Schema | not_exported = {}, config | Schema } in
   module1 = {
     Schema.foo | Number,
     Schema.bar | String,
+    Schema.my_option
+      | { _ : String }
+      | doc m%"
+          The set of things that will wobble up when
+          stuff wiggles down and bubbles sideways
+        "%,
     config | Schema,
     config.bar = "world",
   },

We can now refer to it from module2, and have our editor tell us everything we want to know:

Conclusion

We’ve seen how we can easily implement an equivalent of the NixOS module system on top of Nickel. We’ve also seen how this maps nicely to the semantics of the language and gives us access to all the benefits of good error messages and editor integration.

This is obviously just scratching the surface, and there’s a ton of extra features of NixOS modules that haven’t been covered here. Some would be trivial to implement, like submodules (left as an exercise to the reader). Others would require more work, or even changes to the underlying language like custom types with custom merge functions. However, even this simple formalization is already tremendously powerful. It is used in an experimental branch of Organist to provide a principled way of combining different, independent, but related pieces of functionality.

A great possible follow-up work would be to hook that up with existing NixOS modules (maybe using a mixture of jsonschema-converter and json-schema-to-nickel) to allow writing one’s NixOS configuration directly in Nickel… maybe soon?

Table of contents

• Overview
• Motivation
• Statistics
• Retrospective
• Thoughts on future work
• Acknowledgements

Overview

This is a retrospective of my and many other people’s work on documentation in the Nix ecosystem between October 2022 and March 2024. It serves as a showcase of what we achieved together, and gives an impression of what’s involved in improving the user experience in a complex software system. A lot has happened during that time, so this text is quite lengthy.

The details of this report will mainly be interesting to power users and active contributors, or people working on software projects that are in a similar situation as Nix. For everyone else, I’d summarise the effort so far as “success in slow motion that indicates compounding effects”. Much of it was made possible through ongoing sponsorship from Antithesis, Tweag, and many individuals, along with the incredible commitment of numerous volunteers.

• nix.dev is now the official documentation hub for the Nix ecosystem, designed to follow the Diátaxis documentation framework.
• There are a number of new in-depth tutorials and practical guides, with more in the making.
• The contribution process got smoother and is documented much better.
• The Nix manual substantially improved in terms of structure, clarity, and detail. It can now be accessed by release version on nix.dev.
• The Nixpkgs and NixOS manuals are now entirely written in Markdown, after two years of migrating away from XML.
• More people than ever have contributed improvements to documentation.
• The official NixOS Wiki has launched under wiki.nixos.org.

It may not seem like much, but these are big changes that posed enough of a challenge to arrive at. It’s still not easy or fast to learn all the things needed to wield the full power of Nix. However, there are reports of people grasping most of it within a couple of weeks on their own, which would be a significant improvement over what previously seemed to take months. And as more contributors chime in, things are continuously getting better.

The next steps will be expanding reference documentation, developing more suitable technical infrastructure, adding more tutorials, shaping an information architecture that connects the ecosystem… and simply figuring a lot of things out and writing them down.

Motivation

What originally compelled me to start working on improving Nix documentation was a vague feeling that there was something profound about Nix, something that was merely obscured by confusing explanations. It wasn’t just the fact that the Nix ecosystem permanently solves many problems in computing I believe should never have existed in the first place. It was the question: What underlying principle makes it possible for it to do so? Being in the right place at the right time often enough has allowed me to pursue a few hunches as part of my day job at Tweag for the past two years. And that time was needed.

First, it took me a while to fully grasp how beautifully simple Nix is conceptually:

• The Nix store has a computation engine that operates on file system trees, disguised as a system for caching sandboxed execve calls.
• The Nix language’s most distinguishing features are, not coincidentally, first-class facilities for dealing with file system paths and strings.
• Nixpkgs is the world’s largest knowledge base for getting open source software to run.
• NixOS offers a uniform user interface for configuring that open source software.

Second, after staring at it long enough, it turns out that the underlying powerful idea is programming itself! More precisely, disciplined programming:

• The Nix store enforces referential integrity on the file system and constrains processes to act like pure functions, and it facilitates distributed, incremental computation. Read more on that in Taming Unix with functional programming.
• Nixpkgs is written in the purely functional Nix language, and keeps scaling due to a substantial amount of automation.
• The magic behind NixOS (and related tools, such as Home Manager) is the module system, which combines a rich type system with modular composition. Together with virtual machines, it enables systematic integration testing of a large part of the ecosystem’s artifacts.

From the outside, all of this seems terribly messy. It has grown organically over 20 years, and most of it is still only sparsely documented. Some parts are hard to understand; but that’s often because programming can be fundamentally hard. And it doesn’t help that things are just as messy on the inside. There are places where I’d say “the code doesn’t understand itself”.

The same things that make Nix hard to learn and use also make it hard to document and teach: There are many ways of doing certain things with Nix, and often none of them is clearly superior. A large part of that problem must eventually be solved at the implementation level, but that is subject to the Lisp Curse. In a volunteer-driven open source software community, many technical issues reduce to social issues. The real challenge is coordination!

That’s not exactly what I had originally subscribed to. But in retrospect, maybe it’s not surprising that, apart from finding the right words and the right place for them, the path to better documentation is paved with debates, design documents, fundraisers, UX workshops, governance meetings — that is, listening and talking to lots of people.

Making the ecosystem more approachable for a mass audience may as well end up to be teaching “disciplined software development with Nix”. But even that will require us, as a community-of-practice, to agree on (rather than just find) answers to many open questions. Until then, this report is meant to be a panorama of what that process has entailed so far, from my perspective. I hope that it will encourage you to try for yourself and participate, or otherwise help out.

Statistics

Before getting into a lengthy narrative, let’s look at recorded documentation activities of the past two years. tl;dr: numbers go up, this is good.

The following three charts show the number of documentation-related pull requests merged in the three repositories the team has worked on: Nix, Nixpkgs, and nix.dev. The charts have the same vertical scale and each bar stands for a quarter of a year since the beginning of 2022. Documentation work on Nix is underrepresented prior to Q3, since we only then began systematically labeling those pull requests. For nix.dev all pull requests are counted. The most recent period accounts for at most half of the quarter, because this is when I compiled the data.

Notably, Nixpkgs documentation activities have grown roughly with (and possibly even slower than) the overall number of users and contributors. This sample shows all pull requests automatically labeled as being related to documentation, which often comes with package updates and are often inconsequential in practice.

In contrast, the frequency of changes in nix.dev and the Nix manual greatly increased since the documentation team was founded.

The next three charts show the number of GitHub issues related to documentation that were opened or closed in a given quarter, again for Nix, Nixpkgs, and nix.dev. These charts also have the same scale. Here, one can observe that Nixpkgs documentation activities, while more pronounced overall due to the greater number of people involved, are significantly driven by the team. There is also some up-and-down due to alternating between more exploratory and more implementation-oriented periods.

The final three charts show the number of new contributors and those who made repeated contributions related to documentation, again per quarter and for Nix, Nixpkgs, and nix.dev. These charts are not to scale with each other because including Nixpkgs would have made the others unreadable. This is the most notable change, and in my opinion the most important success: We now have around 10 people working consistently on Nix reference documentation and introductory materials — more than ever!

And now for the full story, which is long because we have made and learned so many things.

Retrospective

In 2022, I started out with the hypothesis that there needs to be a comprehensive document that one could read cover to cover to understand the Nix ecosystem. With Nix inventor Eelco Dolstra’s blessing and Tweag’s funding I started an effort to write what we called The Nix Book.

It turned out not to be that simple at all. The ecosystem is too large, the spectrum of use cases too wide, and the details too messed up to just sit down and blurt out everything one knows, and then wait for others to fix the typos. In fact, there have been multiple such attempts by different people.

A more systematic approach was needed. In addition to an initial literature survey, I also ran a small usability study and evaluated the 2022 community polls. During that time, the documentation team was founded, and became the forum for developing and refining goals for documentation in the Nix ecosystem, and working towards their achievement.

As of today, the documentation team meeting was held more than a hundred times. I’m convinced: we’re delivering. You can feel the change in pace and atmosphere, and the numbers support it.

Over the period covered by this report, we formulated multiple sets of goals. They reflect our evolving understanding of the problem space and shifts in priorities.

Because all other attempts to make a comprehensive tractable review turned out to be impractical, I will first list the goals in the order they were published, for reference. Then, based on the statements reworded and regrouped to avoid overlap, I will present results for each category separately.

Documentation goals for the Nix ecosystem

I chose the following categories for evaluating documentation activities, because they emerge from the concrete goals recorded in the past. I ordered them taking into account their relative importance and the sequence of events; but mostly all of this is merely a narrative device to condense a lot of information.

Improve discoverability

Many of the most obvious issues with Nix-related documentation revolve around obstacles to finding relevant information. We made notable progress on improving discoverability, by more clearly separating the roles of different information sources, reworking navigation paths, and adding cross-references. But we also ran into deeper structural and technical problems, which we started to address from various angles.

Increase coverage of reference documentation

The Nix ecosystem is a large piece of software. The goal is to eventually capture all of its interfaces in reference documentation.

We mainly added much more information to the Nix reference manual, but have not yet started fully documenting the Nix language syntax. There were many incremental improvements to Nixpkgs reference manual, and work is ongoing to further increase coverage. The people involved with the documentation team did not work on the NixOS manual, and the official NixOS Wiki has only recently launched.

Ensure correctness of reference documentation

Merely having interface documentation is not enough, the information also has to be correct and up to date.

Contributors corrected countless errors and omissions in all documentation resources. Work is ongoing to render all of the Nixpkgs library documentation from the source code. There are technical obstacles to increase automatic presentation of interface documentation in the Nix reference manual.

The biggest impact on correctness would be made by automatically testing examples as part of continuous integration, but there was no progress on that apart from exchanging ideas. We’ve been updating examples while working on other things, to make them more self-contained so they could be tested eventually.

Improve the contributor experience

I claim that one of the most important use cases of any community-driven open source project is contributing to that project. Much of our efforts revolved around easing contributions, in particular to documentation. We added and improved contributor guides for each component of the Nix ecosystem, and added automation to further reduce friction. Substantial parts of official documentation are now actively maintained, and the team holds regular meetings to review and merge pull requests. As a consequence, in the second half of 2023 we got much more work done than ever, and more people than ever joined and stayed to help.

Teach important skills

Part of the documentation team’s mission is to provide more and better learning materials. Thanks to many volunteers’ patience and endurance, the group produced roughly one tutorial every two months between end of 2022 and end of 2023. This reportedly improved the learning experience — especially where we took the time to build a solid foundation in reference documentation. That way, we have addressed a good chunk of fundamental use cases.

But there is still much ground to cover, even for what we consider essential skills and workflows in the Nix ecosystem. Some articles could be a lot shorter. Nonetheless, there are now more and much better tutorials than ever, all of which are actively maintained to incorporate feedback and keep up with best practices.

Explain concepts

The Nix ecosystem approaches many problems in a unique way, which requires introducing new terms and explaining why certain things are the way they are. While that was my original motivation to start working on documentation, explanations have shown to be less of an important tool for serving immediate needs. As a result, only a few things were worked on.

Reduce onboarding time

The primary mission of the documentation team is to reduce onboarding time for beginners. That plays into addressing the other challenges I have talked about, such as attracting more contributors. We figured that a sequence of lessons focused on widely applicable skills would be an effective resource for users to become self-sufficient in the ecosystem.

To that end, there was a coordinated project to design what we called the Learning Journey, which was supported by many people through a fundraiser on Open Collective. Despite some setbacks, we achieved the main goals and will continue with implementation by writing more tutorials for the curriculum.

Summary

All in all, we can claim success: The number of new and regular contributors to Nix documentation and nix.dev has increased significantly every quarter. The average turnaround time for documentation contributions seems to have decreased. We’ve come quite far with smoothing out the first couple of days and weeks of using vanilla Nix. Based on feedback and heuristics, such as questions asked on Discourse, I conclude that we have substantially reduced the time required for onboarding with Nix. I’m convinced that discoverability of learning materials and reference documentation has also improved a lot. Some of these results have yet to be validated with more evidence.

Thoughts on future work

I think that Nix documentation is on the right trajectory to eventually shake off its negative reputation and thus help unleash the power of Nix onto the world. I suggest to continue following the goals discussed in this report, and in order to reach them more quickly:

• Finish things that have been started.
• Put more emphasis on technical infrastructure and automation.
• Focus on fewer topics, prioritise reference documentation.
• Cultivate code ownership and empower more people to become maintainers.
• Write more tutorials outlined in the Learning Journey.
• Help establish a consistent information architecture across the Nix ecosystem.
• Secure more funding to support volunteer efforts.
• Systematically measure effectiveness of documentation efforts.

Acknowledgements

This whole endeavor has only been possible to sustain thanks to ongoing financial support. I’d like to thank the leadership of Antithesis, Tweag, Flox, and Determinate Systems for funding me, Silvan, Zach, and Luc to lead and assist the documentation team’s efforts over the past 18 months. Many thanks also to all our supporters on Open Collective for your confidence in our mission and your crucial role in speeding up progress.

An enormous amount of work has also been done by numerous volunteers, who did not just contribute to multiple articles and many small improvements, but were also involved in arriving at technical decisions and implementing them. Some of them stay around to incorporate corrections to their texts and code. Thank you so much for your indispensable contributions to our collective endeavor:

• Alejandro Sánchez Medina (@alejandrosame)
• Lorenzo Manacorda (@asymmetric)
• Alexander Bantyev (@balsoft)
• Bob van der Linden (@bobvanderlinden)
• Bernardo Vecchia Stein (@danielsidhion)
• Domen Kozar (@domenkozar)
• John Ericson (@Ericson2314)
• Henrik Karlsson (@henrik-ch)
• Johannes Kirschbauer (@hsjobeki)
• @lassulus
• Jörg Thalheim (@Mic92)
• Shahar “Dawn” Or (@mightyiam)
• Naïm Favier (@ncfavier)
• Norbert Melzer (@NobbZ)
• Olaf Hochherz (@olafklingt)
• @pennae
• Alexander Groleau (@proofconstruction)
• Ryan Lahfa (@raitobezarius)
• Solène Rapenne (@rapenne-s)
• Robert Hensing (@roberth)
• Ron Efroni (@ronef)
• Jeff Huffman (@tejing1)
• Théophane Hufschmitt (@thufschmitt)
• Thomas Bereknyei (@tomberek)
• Daniel Ramirez (@wamirez)
• Yorick van Pelt (@YorickvP)
• Yuki Langley (@yukiisbored)
• Zach Mitchell (@zmitchell)

Despite its importance in the Nix world, the Nix protocol has no specification or reference documentation. Besides the original implementation in the Nix project itself, the hnix-store project contains a re-implementation of the client end of the protocol. The gorgon project contains a partial re-implementation of the protocol in Rust, but we didn’t know about it when we started. We do not know of any other implementations. (The Tvix project created its own gRPC-based protocol instead of re-implementing a Nix-compatible one.)

So we re-implemented the Nix protocol, in Rust. We started it mainly as a learning exercise, but we’re hoping to do some useful things along the way:

• Document and demystify the protocol. (That’s why we wrote this blog post! 👋)
• Enable new kinds of debugging and observability. (We tested our implementation with a little Nix proxy that transparently forwards the Nix protocol while also writing a log.)
• Empower other third-party Nix clients and servers. (We wrote an experimental tool that acts as a Nix remote builder, but proxies the actual build over the Bazel Remote Execution protocol.)

Unlike the hnix-store re-implementation, we’ve implemented both ends of the protocol. This was really helpful for testing, because it allowed our debugging proxy to verify that a serialization/deserialization round-trip gave us something byte-for-byte identical to the original. And thanks to Rust’s procedural macros and the serde crate, our implementation is declarative, meaning that it also serves as concise documentation of the protocol.

Structure of the Nix protocol

A Nix communication starts with the exchange of a few magic bytes, followed by some version negotiation. Both the client and server maintain compatibility with older versions of the protocol, and they always agree to speak the newest version supported by both.

The main protocol loop is initiated by the client, which sends a “worker op” consisting of an opcode and some data. The server gets to work on carrying out the requested operation. While it does so, it enters a “stderr streaming” mode in which it sends a stream of logging or tracing messages back to the client (which is how Nix’s progress messages make their way to your terminal when you run a nix build). The stream of stderr messages is terminated by a special STDERR_LAST message. After that, the server sends the operation’s result back to the client (if there is one), and waits for the next worker op to come along.

The Nix wire format

Nix’s wire format starts out simple. It has two basic types:

• unsigned 64-bit integers, encoded in little-endian order; and
• byte buffers, written as a length (a 64-bit integer) followed by the bytes in the buffer. If the length of the buffer is not a multiple of 8, it is zero-padded to a multiple of 8 bytes. Strings on the wire are just byte buffers, with no specific encoding.

Compound types are built up in terms of these two pieces:

• Variable-length collections like lists, sets, or maps are represented by the number of elements they contain (as a 64-bit integer) followed by their contents.
• Product types (i.e. structs) are represented by listing out their fields one-by-one.
• Sum types (i.e. unions) are serialized with a tag followed by the contents.

For example, a “valid path info” consists of a deriver (a byte buffer), a hash (a byte buffer), a set of references (a sequence of byte buffers), a registration time (an integer), a nar size (an integer), a boolean (represented as an integer in the protocol), a set of signatures (a sequence of byte buffers), and finally a content address (a byte buffer). On the wire, it looks like:

3c 00 00 00 00 00 00 00 2f 6e 69 78 2f 73 74 6f 72 65 ... 2e 64 72 76 00 00 00 00  <- deriver
╰──── length (60) ────╯ ╰─── /nix/store/c3fh...-hello-2.12.1.drv ───╯ ╰ padding ╯

40 00 00 00 00 00 00 00 66 39 39 31 35 63 38 37 36 32 ... 30 33 38 32 39 30 38 66  <- hash
╰──── length (64) ────╯ ╰───────────────────── sha256 hash ─────────────────────╯

02 00 00 00 00 00 00 00                                                            ╮
╰── # elements (2) ───╯                                                            │
                                                                                   │
   39 00 00 00 00 00 00 00 2f 6e 69 78 ... 2d 32 2e 33 38 2d 32 37 00 00 .. 00 00  │
   ╰──── length (57) ────╯ ╰── /nix/store/9y8p...glibc-2.38-27 ──╯ ╰─ padding ──╯  │ references
                                                                                   │
   38 00 00 00 00 00 00 00 2f 6e 69 78 ... 2d 68 65 6c 6c 6f 2d 32 2e 31 32 2e 31  │
   ╰──── length (56) ────╯ ╰───────── /nix/store/zhl0...hello-2.12.1 ───────────╯  ╯

1c db e8 65 00 00 00 00 f8 74 03 00 00 00 00 00 00 00 00 00 00 00 00 00            <- numbers
╰ 2024-03-06 21:07:40 ╯ ╰─ 226552 (nar size) ─╯ ╰─────── false ───────╯

01 00 00 00 00 00 00 00                                                            ╮
╰── # elements (1) ───╯                                                            │
                                                                                   │ signatures
   6a 00 00 00 00 00 00 00 63 61 63 68 65 2e 6e 69 ... 51 3d 3d 00 00 00 00 00 00  │
   ╰──── length (106) ───╯ ╰─── cache.nixos.org-1:a7...oBQ== ────╯ ╰─ padding ──╯  ╯

00 00 00 00 00 00 00 00                                                            <- content address
╰──── length (0) ─────╯

This wire format is not self-describing: in order to read it, you need to know in advance which data-type you’re expecting. If you get confused or misaligned somehow, you’ll end up reading complete garbage. In my experience, this usually leads to reading a “length” field that isn’t actually a length, followed by an attempt to allocate exabytes of memory. For example, suppose we were trying to read the “valid path info” written above, but we were expecting it to be a “valid path info with path,” which is the same as a valid path info except that it has an extra path at the beginning. We’d misinterpret /nix/store/c3f-...-hello-2.12.1.drv as the path, we’d misinterpret the hash as the deriver, we’d misinterpret the number of references (2) as the number of bytes in the hash, and we’d misinterpret the length of the first reference as the hash’s data. Finally, we’d interpret /nix/sto as a 64-bit integer and promptly crash as we allocate space for more than $8 \times 10^{18}$ references.

There’s one important exception to the main wire format: “framed data”. Some worker ops need to transfer source trees or build artifacts that are too large to comfortably fit in memory; these large chunks of data need to be handled differently than the rest of the protocol. Specifically, they’re transmitted as a sequence of length-delimited byte buffers, the idea being that you can read one buffer at a time, and stream it back out or write it to disk before reading the next one. Two features make this framed data unusual: the sequence of buffers are terminated by an empty buffer instead of being length-delimited like most of the protocol, and the individual buffers are not padded out to a multiple of 8 bytes.

Serde

Serde is the de-facto standard for serialization and deserialization in Rust. It defines an interface between serialization formats (like JSON, or the Nix wire protocol) on the one hand and serializable data types on the other. This divides our work into two parts: first, we implement the serialization format, by specifying the correspondence between Serde’s data model and the Nix wire format we described above. Then we describe how the Nix protocol’s messages map to the Serde data model.

The best part about using Serde for this task is that the second step becomes straightforward and completely declarative. For example, the AddToStore worker op is implemented like

#[derive(serde::Deserialize, serde::Serialize)]
pub struct AddToStore {
    pub name: StorePath,
    pub cam_str: StorePath,
    pub refs: StorePathSet,
    pub repair: bool,
    pub data: FramedData,
}

These few lines handle both serialization and deserialization of the AddToStore worker op, while ensuring that they remain in-sync.

Mismatches with the Serde data model

While Serde gives us some useful tools and shortcuts, it isn’t a perfect fit for our case. For a start, we don’t benefit much from one of Serde’s most important benefits: the decoupling between serialization formats and serializable data types. We’re interested in a specific serialization format (the Nix wire format) and a specific collection of data types (the ones used in the Nix protocol); we don’t gain much by being able to, say, serialize the Nix protocol to JSON.

The main disadvantage of using Serde is that we need to match the Nix protocol to Serde’s data model. Most things match fairly well; Serde has native support for integers, byte buffers, sequences, and structs. But there were a few mismatches that we had to work around:

• Different kinds of sequences: Serde has native support for sequences, and it can support sequences that are either length-delimited or not. However, Serde does not make it easy to support length-delimited and non-length-delimited sequences in the same serialization format. And although most sequences in the Nix format are length-delimited, the sequence of chunks in a framed source are not. We hacked around this restriction by treating a framed source not as a sequence but as a tuple with $2^{64}$ elements, relying on the fact that Serde doesn’t care if you terminate a tuple early.
• The Serde data model is larger than the Nix protocol needs; for example, it supports floating point numbers, and integers of different sizes and signedness. Our Serde de/serializer raises an error at runtime if it encounters any of these data types. Our Nix protocol implementation avoids these forbidden data types, but the Serde abstraction between the serializer and the data types means that any mistakes will not be caught at compile time.
• Sum types tagged with integers: Serde has native support for tagged unions, but it assumes that they’re tagged with either the variant name (i.e. a string) or the variant’s index within a list of all possible variants. The Nix protocol uses numeric tags, but we can’t just use the variant’s index: we need to specify specific tags for specific variants, to match the ones used by Nix. We solved this by using our own derive macro for tagged unions. Instead of using Serde’s native unions, we map a union to a Serde tuple consisting of a tag followed by its payload.

But with these mismatches resolved, our final definition of the Nix protocol is fully declarative and pretty straightforward:

#[derive(TaggedSerde)]
//       ^^ our custom procedural macro for unions tagged with integers
pub enum WorkerOp {
    #[tagged_serde = 1]
    //              ^^ this op has opcode 1
    IsValidPath(StorePath, Resp<bool>),
    //             ^^            ^^ the op's response type
    //             || the op's payload
    #[tagged_serde = 6]
    QueryReferrers(StorePath, Resp<StorePathSet>),
    #[tagged_serde = 7]
    AddToStore(AddToStore, Resp<ValidPathInfoWithPath>),
    #[tagged_serde = 9]
    BuildPaths(BuildPaths, Resp<u64>),
    #[tagged_serde = 10]
    EnsurePath(StorePath, Resp<u64>),
    #[tagged_serde = 11]
    AddTempRoot(StorePath, Resp<u64>),
    #[tagged_serde = 14]
    FindRoots((), Resp<FindRootsResponse>),
    // ... another dozen or so ops
}

Next steps

Our implementation is still a work in progress; most notably the API needs a lot of polish. It also only supports protocol version 34, meaning it cannot interact with old Nix implementations (before 2.8.0, which was released in 2022) and will lack support for features introduced in newer versions of the protocol.

Since in its current state our Nix protocol implementation can already do some useful things, we’ve made the crate available on crates.io. If you have a use-case that isn’t supported yet, let us know! We’re still trying to figure out what can be done with this.

In the meantime, now that we can handle the Nix remote protocol itself we’ve shifted our experimental hacking over to integrating with Bazel remote execution. We’re writing a program that presents itself as a Nix remote builder, but instead of executing the builds itself it sends them via the Bazel Remote Execution API to some other build infrastructure. And then when the build is done, our program sends it back to the requester as though it were just a normal Nix remote builder.

But that’s just our plan, and we think there must be more applications of this. If you could speak the Nix remote protocol, what would you do with it?

If you’ve never heard of cloud native computing before, it has a number of definitions online, but the simplest one is that it’s mostly about Kubernetes.

Kubecon is a huge event with thousands of attendees. The conference spanned several levels of the main convention center in Paris, with a myriad of conference rooms and a whole floor for sponsor booths. FOSDEM already felt huge compared to academic conferences, but Kubecon is even bigger.

Although the program was filled with appealing talks, we ended up spending most of our time chatting with people and visiting booths, something you can’t do as easily online.

Nix for the win

The very first morning, as we were walking around and waiting for the caffeine to kick in, we immediately spotted a Nix logo on someone’s sweatshirt. No better way to start the day than to meet with fellow Nix users!

And Nix was in general a great entry point for conversations at this year’s Kubecon. We expected it to still be an outsider at an event about container-driven technology, but the problems that “containers as a default packaging unit” can’t solve were so present that Nix’s value proposition is attuned to what everyone had on their minds anyway. In other words, Nix is now known enough as to serve as a conversation starter: the company might not use it, but many people have heard about it and were very interested to hear insights from big contributors like Tweag, the Modus Create OSPO.

Cybersecurity

Many security products were represented. Securing cloud-native applications is indeed a difficult matter, as their many layers and components expose a large attack surface.

For one, the schedule included a security track, with a fair share of talks being about SBOMs tying back into the problem of containers that ship an opaque content without inventory. Nix, as a solution to this problem, was a great conversation starter here as well, especially for fellow Nixer Matthias, who can talk for hours about how Nix is the best (and maybe only) technology for automatically deriving complete SBOMs of a piece of software, in a trustworthy manner. Our own NLNet-funded project genealogos, which does exactly that, is recently getting a lot of interest.

Besides the application code and what goes in it, another focus was avoiding misconfiguration of the cloud infrastructure layer, of the Kubernetes cluster, and anything else going into container images. Many companies propose SaaS combinations of static linters scanning the configuration files directly with various policy rules, heuristics and dynamic monitoring of secure cloud native applications. Our configuration language Nickel was very relevant here: one of its raisons d’être is to provide efficient tools (types, contracts and a powerful LSP) to detect and fix misconfigurations as early as possible. We had cool conversations around writing custom security policies as Nickel contracts with the new LSP background contract checking (introduced in 1.5) reporting non-compliance live in the editor — in that light, contracts are basically a lightweight way to program an LSP.

Internal Developer Platforms (IDPs)

IDPs were a hot topic at Kubecon. Tweag’s mission, as the OSPO of a leading software development consultancy, is to improve developer experience across the software lifecycle, which makes IDPs a natural topic for us.

An IDP is a platform - usually a web interface in practice - which glues several developer tools and services together and acts as the central entry point for most developer workflows. It’s centered around the idea of self-service, not unlike the console of cloud providers, but configurable for your exact use case and open to integrate tools across ecosystem boundaries. We already emphasized this emerging new abstraction in last year’s post. Example use cases are routinely deploying new infrastructure with just a few clicks, rather than requiring to sync and to send messages back-and-forth to the DevOps team. IDPs don’t replace other tools but offer a unified interface, usually with customized presets, to interact with repositories, internal data and infrastructure.

Backstage, an open-source IDP developed by Spotify, had its own sub-conference at Kubecon. Several products are built on top of it as well: it’s not really a ready-to-use solution but rather the engine to build a custom IDP for your company, which leaves room for turnkey offers. We feel that such integrated, centralized and simple-to-use services may become a standard in the future: think of how much GitHub (or an equivalent) is a central part of our modern workflow, but also of how many things it frustratingly can’t do (in particular infrastructure).

Cloud & AI

Many Kubernetes-based MLOps companies propose services to make it easy to deploy and manage scalable AI models in the cloud.

In the other direction, with the advent of generative AI, there was no doubt that we would see AI-based products to ease the automation of infrastructure-related tasks. We attended a demo of a multi-agent system which integrates with e.g. Slack, where you can ask a bot to perform end-to-end tasks (which includes interacting with several systems, like deploying something to the cloud, editing a Jira ticket and pushing something to a GitHub repo) or ask high-level questions, such as “which AWS users don’t have MFA enabled”.

It’s hard to tell from a demo how solid this would be in a real production system. I also don’t know if I would trust an AI agent to perform tasks without proper validation from a human (though there is a mode where confirmation is required before applying changes). There are also security concerns around having those agents run somewhere with write access to your infrastructure.

Putting those important questions aside, the demo was still quite impressive. It makes sense to automate those small boring tasks which usually require you to manually interact with several different platforms and are often quite mechanical indeed.

OSPO

We attended a Birds-of-a-Feather session on Open Source Program Offices (OSPO). While the small number of participants was a bit disappointing (between 10 and 15, compared to the size of the conference), the small group discussions were still engrossing, and we were pleased to meet people from other OSPOs as well as engineers wanting to push for an OSPO in their own company.

The generally small size OSPOs (including from very large and influential tech companies) and their low maturity from a strategic point of view was surprising to us. Many OSPOs seem to be stuck in tactical concerns, managing license and IP issues that can occur when developers open up company-owned repos. In such a situation, all OSPO members are fully occupied by the large number of requests they get. But the most interesting questions: how to share benefits and costs by working efficiently with open source communities, how to provide strategic guidance and support, and how to gain visibility in communities of interest were only addressed by few. A general concern seemed to be generally a lack of understanding by upper management about the real strategic power that an OSPO can provide. From that perspective, Tweag, although a pink unicorn as a consulting OSPO, is quite far on the maturity curve with concrete strategical and technical firepower through technical groups, and our open-source portfolio (plus the projects that we contribute to but aren’t ours).

Concluding words

Kubecon was a great experience, and we’re looking forward to the next one. We are excited about the advent of Internal Developer Platforms and the concept of self-serving infrastructure, which are important aspects of developer experience.

On the technological side, the cloud-native world seems to be dominated by Kubernetes with Helm charts and YAML, and Docker, while the technologies we believe in and are actively developing are still outsiders in the space (of course they aren’t a full replacement for what currently exists, but they could fill many gaps). I’m thinking in particular about Nix (and more generally about declarative, hermetic and reproducible builds and deployments) and Nickel (better configuration languages and management tools). But, conversation after conversation, conference after conference, we’re seeing more and more interests in new paradigms, sometimes because those technologies are best equipped - by far - to solve problems that are on everyone’s radar (e.g. software traceability through SBOMs with Nix) thanks to their different approach.

Dear CISA team,

I appreciate your effort to gather comments about your recently released “Software Identification Ecosystem Option Analysis” white paper. As you say in the Executive Summary, “Organizations of all sizes must track what software they own and operate to perform user support, inventory administration, and vulnerability management”. I would go further and claim that for any software system that will be modified and reassembled — which is basically always — precise knowledge and control over the components and how they should be put together is crucial. Precise naming, identifiers, are the basis of that. In that light, I would like to bring to your attention another noteworthy technology that hasn’t been mentioned in this study, Nix and its sister project Guix, that achieves exactly this.

Nix is a powerful package manager, offering a very distinctive approach to deploying software. It achieves very high levels of reproducibility and provenance tracking of software artifacts by design, utilizing a functional and declarative language to describe software builds and their dependencies.

In fact, the levels of reproducibility it achieves are so high that Nix can robustly rely on an “input-addressed” storage, an identification model that names software artifacts by hashing everything required to build them, as opposed to hashing their content once assembled. This unique input-addressed approach is very powerful because it allows computing the identifier of a software asset without assembling it.

“Software artifacts” in Nix can be anything from sources and data assets to executable binary packages. And, importantly, “everything required to build” is not limited to source code and data assets, as in most intrinsic identification models, but comprises all commands, configuration, and recursively identified dependencies that are required to assemble the asset in a very strict sandbox. This controlled environment ensures that the description and the identifier of a software asset (called “package closure” in Nix) are complete, capturing all ingredients that went into the final output.

The reproducibility of core packages of the Nix distribution NixOS is very high, automatically tested, and can in principle be used for the independent, and decentralized verification of the content of software artifacts, as demonstrated by this implementation developed within the European Commission’s Next Generation Internet program.

In addition, these advantageous properties allowed the Nix community to construct Nixpkgs, the largest, and most up to date, open software library available. As required by the Nix model, this enormous repository of software assets comprises not only a description of the components that have been used to assemble them but also everything else necessary to produce the final packages, including, besides build instructions and configuration, a global dependency graph of all these assets. And, besides the guarantees that Nix furnishes by design for completeness and reproducibility, the packages go through automatic tests executed by an associated CI, and the hands of tens of thousands of regular users.

The screenshot below shows the CI build example of the open source computer game EmptyEpsilon. Omitting some details in the screenshot below, rja769qkxhiha7mbhq5bjmkjd0d5l1v0-empty-epsilon-2023.06.17 in the Derivation store path field, is the unique identifier of the particular version of this software package realized in the context of all the dependencies and configuration that are defined in a specific version (commit) of the Nixpkgs library that this CI build is attached to. “release.nix” is the entrypoint into the Nixpkgs library of software assets where EmptyEpsilon is defined using the Nix language; the mentioned .drv file contains an intermediate, raw build recipe that was generated from the Nix expressions. Software assets can come with attached metadata such as license information or short descriptions, and some data such as the closure size comprising the software and all its dependencies (build- and run-time) can easily be computed. More details on this can be found here or here.

The exceptional completeness, robustness and precision of this approach uniquely positions Nix (and Guix) as highly valuable tool for automatically creating and managing accurate, reliable and reproducible software bills of materials (SBOMs) that can be employed to address the challenges outlined in the CISA notice. In fact, several projects exist (e.g. 1,2,3) that aim to automatically generate SBOMs from Nix expressions, or connect Nix packages to NIST’s national vulnerability database. The Nix model for identifiers works very well together with SoftWare Hash IDentifiers (SWHID) for the full state of version control system repositories that are developed by Software Heritage.

Finally, and certainly most importantly, I would like to emphasize that tens of thousands of users are demonstrating every day that applying this model comes without overhead. In fact, the precision and robustness of these software identifiers comes with a multitude of additional benefits. All this is not magic but enabled by a tool that follows a rigorous deployment model, the output of decades of academic experimentation and research. This is why the user bases of Nix and Guix, still emerging technologies, are rapidly growing.

However, rules_nixpkgs is incompatible with remote execution. This is a major limitation given that remote execution is possibly the main reason why people switch to Bazel. And that rules_nixpkgs provides a great way to configure hermetic toolchains, which are an important ingredient for reliable remote execution. There is no trivial fix as can be seen in the related, longstanding open issue. At Tweag we investigated a promising solution presented at Bazel eXchange 2022 (recording), but these ideas were never implemented in a public proof of concept.

In this post, we will present our new remote execution infrastructure repo and walk you through the required steps to comprehend and replicate how it achieves remote execution with rules_nixpkgs.

The remote execution limitation

When we make use of rules_nixpkgs, we instruct Bazel to use packages from nixpkgs rather than those from the host system. This means that when we try to build a C++ project, Bazel won’t use the gcc compiler, which is typically found under /usr/bin, but instead will use the compiler specified by rules_nixpkgs and provided by Nix, typically stored under some /nix/store/<unique_hash>-gcc/bin directory.

Bazel distinguishes actions to import external dependencies from regular build actions. The former are always executed locally¹, while the latter can be distributed using remote execution. rules_nixpkgs falls into the former category and invokes Nix to download and install the required /nix/store/<unique_hash>-gcc path locally on your machine.

This scenario works fine when we’re building locally. However, when we enable remote execution, rules_nixpkgs still installs dependencies locally, while the build happens on another machine, which will not have those paths available, so it will inevitably fail.

Initial setup with remote execution

For our proof of concept, we decided to use Buildbarn to provide the remote execution endpoint and infrastructure. Buildbarn provides Kubernetes manifests that we can use to deploy all the necessary Buildbarn components for remote execution to work. We’ll be using the examples from the bb-deployments repository to test our setup, but also modifying it to make use of rules_nixpkgs.

To replicate our implementation you’ll need a working Buildbarn infrastructure, which in this case would be a Kubernetes cluster. You can use our guide to set up a cluster on AWS.

Test remote execution without rules_nixpkgs

To make sure that everything is working as expected, we’ll use the @abseil-hello Bazel target which is available in the Buildbarn deployments repo. This example does not use rules_nixpkgs, yet. You can clone the bb-deployments repository, if you want to follow along.

• Get the service endpoint of the Buildbarn executor service (frontend). If you’re deploying on a cloud provider this would be a load-balancer.

$ kubectl get services -n buildbarn
NAME        TYPE           CLUSTER-IP      EXTERNAL-IP                         PORT(S)                      AGE
browser     ClusterIP      172.20.22.171   <none>                              7984/TCP                     8d
frontend    LoadBalancer   172.20.126.97   xxxxx.us-east-1.elb.amazonaws.com   8980:31657/TCP               8d
scheduler   ClusterIP      172.20.83.110   <none>                              8982/TCP,8983/TCP,7982/TCP   8d
storage     ClusterIP      None            <none>                              8981/TCP                     8d

• Update .bazelrc to use the remote executor endpoint of our environment

...
build:remote-exec --remote_executor=grpc://[endpoint-from-previous-step]
...

Now we can try building the @abseil-hello target using the remote execution infrastructure. Note that we’ll be using a custom toolchain specific to the default executors created by Buildbarn.

bazel build --config=remote-ubuntu-22-04 @abseil-hello//:hello_main

Test remote execution with rules_nixpkgs

Once we have validated that our setup works we can create a new target that uses rules_nixpkgs.

Update .bazelversion to use 6.4 which is a version supported by rules_nixpkgs (any other version on the 6.x should work as well).

Update the WORKSPACE file with the following:

http_archive(
    name = "io_tweag_rules_nixpkgs",
    strip_prefix = "rules_nixpkgs-244ae504d3f25534f6d3877ede4ee50e744a5234",
    urls = ["https://github.com/tweag/rules_nixpkgs/archive/244ae504d3f25534f6d3877ede4ee50e744a5234.tar.gz"],
)

load("@io_tweag_rules_nixpkgs//nixpkgs:repositories.bzl", "rules_nixpkgs_dependencies")
rules_nixpkgs_dependencies()

load("@io_tweag_rules_nixpkgs//nixpkgs:nixpkgs.bzl", "nixpkgs_git_repository", "nixpkgs_package", "nixpkgs_cc_configure")

load("@io_tweag_rules_nixpkgs//nixpkgs:toolchains/go.bzl", "nixpkgs_go_configure") # optional

nixpkgs_git_repository(
    name = "nixpkgs",
    revision = "23.11",
)

nixpkgs_cc_configure(
  repository = "@nixpkgs",
  name = "nixpkgs_config_cc",
  attribute_path = "clang",
)

This is the standard boilerplate to install rules_nixpkgs on our Bazel workspace. We’re also creating a reference to the nixpkgs repository, and a C++ toolchain using clang.

Next, we create a new cc_binary target in BUILD.bazel with a simple hello-world program.

$ cat BUILD.bazel
...
cc_binary(
    name = "hello-world",
    srcs = ["hello-world.cc"],
)

$ cat hello-world.cc
#include <iostream>

int main(int argc, char** argv) {
  std::cout << "Hello world!" << std::endl;
  return 0;
}

Now we need to update the custom Buildbarn toolchain used by the executors to reference @nixpkgs_config_cc. Update the file tools/remote-toolchains/BUILD.bazel and replace the instances of @remote_config_cc with @nixpkgs_config_cc.

We can try building the application using the C++ toolchain we defined with rules_nixpkgs. We expect this to fail because the executors are not Nix-aware yet.

$ bazel build --config=remote-ubuntu-22-04 @abseil-hello//:hello_main

...
ERROR: /home/user/.cache/bazel/_bazel_user/5ce2ca33a49034ed7557e24d70204ce5/external/com_google_absl/absl/base/BUILD.bazel:324:11: Compiling absl/base/internal/throw_delegate.cc failed: (Exit 34): Remote Execution Failure:
Invalid Argument: Failed to run command: Failed to start process: fork/exec /nix/store/n37gxbg343hxin3wdryx092mz2dkafy8-clang-wrapper-16.0.6/bin/cc: no such file or directory
...

Because the executors don’t have the /nix/store available, they cannot resolve the compiler path which is generated locally on our machine when we invoke bazel build.

Now let’s see how we can solve this problem by configuring the executors to access a shared /nix/store via NFS.

NFS-based solution

Our solution involves a Nix server that bridges this gap. This server manages and synchronizes the Nix dependencies across the Bazel build environment.

Here’s how it works:

1. During bazel build the rules_nixpkgs repository rules will build and copy any Nix derivation to the remote Nix server.

2. The Nix server will export the /nix/store directory tree via a read-only NFS mount share to the executors.

3. When a build is triggered, all necessary dependencies are already available on the executors, allowing for the build process to continue.

Implementation-wise, we’ll need to make the following changes to the Buildbarn infrastructure:

• A Nix server. This could be a VM with Nix installed that is exporting the /nix/store directory as a read-only NFS share over the private network. We’ll need SSH access on that server from the machine that invokes bazel build.

• Kubernetes executors with the exported NFS share mounted.

For a detailed setup guide and implementation specifics, refer to our infrastructure repository.

To instruct rules_nixpkgs to copy the nix derivations to the server we’ll need to create an entry in our SSH config (typically found under ~/.ssh/config) with the remote server and then set the environment variable BAZEL_NIX_REMOTE with the name of that entry.

# SSH Configuration
$ cat ~/.ssh/config
Host nix-server
  Hostname [public-ip]
  IdentityFile [ssh-private-key]
  Port [ssh-port]
  User [ssh-user]

Testing out remote execution again

With the new setup, we can try building the project again.

$ export BAZEL_NIX_REMOTE=nix-server
$ bazel clean --expunge # To refetch the Nix derivations
$ bazel build --config=remote-ubuntu-22-04 @abseil-hello//:hello_main

You should now see lines like the following, confirming communication with the Nix server

...
Analyzing: target @abseil-hello//:hello_main (0 packages loaded, 0 targets configured)
    Fetching repository @nixpkgs_config_cc_info; Remote-building Nix derivation 9s
...

And the build should be successful.

Conclusion

In this post, we explored the challenges and our solution for integrating rules_nixpkgs with remote execution in Bazel. Of course this solution is not perfect and it comes with some shortcomings that end user should be aware of.

• The first issue is about cache eviction. Caching all the Nix paths over the long term is not practical from a storage standpoint. That’s why we need a way to mark the required paths, and garbage collect the others. A Nix path should be available as long as a client may trigger a remote build that uses it. However, there’s no way to determine when a client no longer needs a specific path. A simple solution will be to invalidate the least used paths. That will require a tighter integration with the Bazel APIs in order to track the Nix path usage.

• The second issue relates to NFS performance. This depends on the infrastructure and workloads in operation. At least we want to tune the NFS synchronization to the point that the paths are available before any build begins. Slow synchronization between the NFS server and client can lead to failed builds.

This post requires some familiarity with Nix and its language. So if you don’t know what Nix is yet, take a look first, it’s pretty neat.

In this post we’re going to look at what source filtering is, why it’s useful, why a new library was needed for it, and the basics of the new library.

This post notably won’t really teach you a lot about the new library, that’s what the official tutorial and the reference documentation is for. But if you’d like to know some background and motivation, please read on!

Why filter sources

You most likely have come across this pattern:

stdenv.mkDerivation {
  src = ./.;
  # stuff..
}

This is the basis for a Nix expression to build a project in a local directory. There’s a lot of magic to make this work, but we’ll focus on just a few aspects:

• Bar some exceptions, attributes passed to stdenv.mkDerivation are automatically turned into environment variables that are available to the derivation builder.
• The relative path expression ./. is turned into a string of the form "/nix/store/<hash>-<name>" by hashing the contents of the directory that Nix file is in, and adding it to the Nix store.

We then end up with a store derivation whose environment variables include src = "/nix/store/<hash>-<name>". To get more info on how all of this works, see the documentation on derivations.

This generally does work, with the big caveat that all files in the local directory are copied into the Nix store and become a dependency of the derivation. This means that:

• Changing any file will cause the resulting derivation to change, making Nix unable to reuse previous build results.

  For example, if you just format your Nix files, Nix will have to build the project’s derivation again!

• If you have secret files in the local directory, they get added to the Nix store too, making them readable by any user on the system!

  Note If you use the experimental new nix CLI with Flakes and Git in a current (2.18.1) version of Nix, only files tracked by Git will be available to Nix, so this generally won’t be a problem.

  But be careful: If you don’t use Git, the entire directory is always copied to the Nix store! Furthermore, experimental features may change over time.

The hardcore way to filter sources

To address this, Nix comes with builtins.path, which allows controlling how paths get added to the Nix store:

builtins.path {
  # The path to add to the store
  path = ./.;
  # A function determining whether to include specific paths under ./.
  filter =
    # A path under ./.
    path:
    # The path type, either "directory", "normal", "symlink" or "unknown"
    type:
    # The return value of this function indicates
    # whether the path should be included or not.
    # In this case we always return true, meaning everything is included
    true;
}

While this interface looks straightforward, it’s notoriously tricky to get it to do what you want.

Let’s give it a try and start with something trivial, say only including a single file in the current directory:

# default.nix
builtins.path {
  path = ./.;
  filter =
    path: type:
    path == ./file;
}

$ touch file

$ tree "$(nix eval --raw -f.)"
/nix/store/dg5zq00kxabc3lfg03bnrfwax1ndgn6s-filter

0 directories, 0 files

It doesn’t work, we just get an empty directory! The problem here is that the filter function is not called with path values but rather strings, and the == operator always returns false when given two values of different types.

To fix this, we can use the builtin toString function, which converts path values to strings:

# default.nix
builtins.path {
  path = ./.;
  filter =
    path: type:
    path == toString ./file;
}

$ tree "$(nix eval --raw -f.)"
/nix/store/2myzf03ca2ch3lc40p7frvqqbvm5nm2m-filter
└── file

1 directory, 1 file

Great, this works! Now let’s try the same for dir/file:

# default.nix
builtins.path {
  path = ./.;
  filter =
    path: type:
    path == toString ./dir/file;
}

$ mkdir dir && touch dir/file

$ tree "$(nix eval --raw -f.)"
/nix/store/dg5zq00kxabc3lfg03bnrfwax1ndgn6s-filter

0 directories, 0 files

Apparently this doesn’t work for nested directories.

The problem now is that the filter function first gets called on dir itself. And because ./dir != ./dir/file, it returns false, therefore excluding dir entirely.

To fix this we need to make sure the function recurses into the directories, which we can do by checking for type == "directory":

# default.nix
builtins.path {
  path = ./.;
  filter =
    path: type:
    # Return true for all directories
    type == "directory"
    # But also for the file we want to include
    || path == toString ./dir/file;
}

But what if there is another directory we don’t care about?

$ mkdir another-dir

$ tree $(nix eval --raw -f.)
/nix/store/wj0y4f4x5llzz8kj48jd0gvszddp3jr0-filter
├── another-dir
└── dir
    └── file

3 directories, 1 file

This worked, dir/file is there. But so is the another-dir, although it doesn’t even contain any files!

We could go on like that, but I think you get the gist: This function is tricky to use!

Introducing the file set library

Let’s compare this to the new file set library:

{ lib ? import (fetchTarball "channel:nixos-23.11" + "/lib") }:
lib.fileset.toSource {
  root = ./.;
  fileset = ./dir/file;
}

$ tree $(nix eval --raw -f. outPath)
/nix/store/csgp388b3zqxp2av01gjncy9sadxib9q-source
└── dir
    └── file

2 directories, 1 file

This is much more straightforward and does what you’d expect. But where is the file set here?

The key here is that when the file set library expects to get a file set, but gets a path, it implicitly turns the path into a file set. So to the library, ./dir/file is a file set containing just the single file ./dir/file, while ./dir would be a file set containing all files in the directory ./dir.

The real power of this library however comes from the fact that file sets behave just like mathematical sets, and it comes with some core functions to support that:

• lib.fileset.union :: FileSet -> FileSet -> FileSet
• lib.fileset.intersection :: FileSet -> FileSet -> FileSet
• lib.fileset.difference :: FileSet -> FileSet -> FileSet

Some other notable features are:

• Files are never added to the store unless explicitly requested with lib.fileset.toSource.
• Maximum laziness: Directories are never recursed into more than necessary.
• Actionable error messages in case something doesn’t look right.
• Minimal assumptions: The library only relies on stable Nix features. It even works correctly with possible future changes to the behavior of Nix paths.

But this is not the best place to teach you about the library. For that, head over to the official tutorial instead, or check out the reference documentation!

The file set library is going to be included in the upcoming NixOS 23.11 release. If you encounter any problems using it or are missing some feature, let me know in this tracking issue.

Comparison

For completeness, we also need to look at previous related efforts and see how they compare to this library:

• builtins.fetchGit ./. allows creating a store path from all Git-tracked files, so it’s very similar to lib.fileset.gitTracked. However, it’s tricky to further restrict or extend the set selected files, since the above filter-based approach wouldn’t work on store paths without some changes.

• lib.sources.cleanSourceWith is a simple wrapper around builtins.path. While it has the same filter-based interface, it improves over builtins.path by being chainable, allowing the set of included files to be further restricted.

• lib.sources.cleanSource uses cleanSourceWith underneath to set filter to a reasonable default, filtering out some of the most common unwanted files automatically. The file set library doesn’t yet have a good replacement for this, but there is lib.fileset.fromSource, which you can use to convert any lib.sources-based value to a file set.

• lib.sources.sourceByRegex and lib.sources.sourceFilesBySuffices are also functions built on top of cleanSourceWith, and as such can be chained with each other. While sourceFilesBySuffices is not bad, the interface of sourceByRegex is rather clunky and error-prone. Furthermore, it’s hard to add more files to the result.

• gitignore.nix and pkgs.nix-gitignore allow you to filter files based on Git’s .gitignore files, which is very related to Git-tracked files. The file set library doesn’t replace these functions, but it can be used as a more composable foundation.

• nix-filter is a third-party lib.sources wrapper. It wraps it with a nicer interface, but suffers from some unclear semantics and composability issues. The file set library should serve as an improved replacement.

• Source combinators was a previous attempt to create a composable interface for handling source files. It was a bit tricky to use and never merged, but this is in fact the work that inspired the new file set library!

Conclusion

We’ve seen that filtering sources can improve your Nix experience by avoiding unnecessary derivation rebuilds. While it was possible to filter sources before using the builtins.path function and other approaches, there are many pitfalls. The lib.fileset library in comparison makes source filtering a breeze.

In addition to a huge thanks to Antithesis as the main sponsor of this work, I’d also like to thank Robert Hensing from Hercules CI and Valentin Gagarin from Tweag for all the help they’ve given me during reviews!

A mess of cables and knobs

I used to play piano as a kid. As a teenager, I became frustrated by the limitations of the instrument and started getting into synthesizers. That was a thrilling experience. Being able to turn a few knobs and get a totally different — and brand new — sound was an amazing feeling. But it was also frustrating. There are so many ways in which you can tune such an instrument (including so many ways of getting an awful sound out of it) that I ended up spending all of my time trying to come up with the right sound, and I didn’t have any left to actually practice the instrument. Besides, the sheer complexity and intricacy of the configuration meant that it was incredibly hard to remember how to get what I wanted out of it.

Then I started playing the organ. Organs might be impressive and daunting, but compared to a synthesizer, they are pretty boring. Modulo a few exceptions, the only controls you have in front of you are the stop knobs which allow you to turn a stop (sound) on or off. That still leaves quite a few possible combinations, but very far from the virtually infinite possibilities of an analog synthesizer. Now I had an expressive enough instrument, but one that knew how to get out of my way to let me play it rather than play with it.

Modern development projects are synthesizer-like these days. We combine a plethora of tools, each one coming with its own set of knobs. Like the old school modular synthesizers, they generally all come from different places and plugging them together requires a huge mess of ad-hoc cabling, like in the good old time!

Organist

It would be tempting to just ditch all of that complexity and only provide a hard-coded set of tools, with hard-coded settings and try to shoehorn everything into that setup. But we all know that such a thing is a fool’s errand. The variety of tools and knobs merely reflects the complexity of the real world, and we have to deal with it one way or another.

Instead, we can limit the scope of that complexity: provide the best possible defaults, and a framework to make it easy to override them in a tractable manner.

Organist is a tool meant to do just that: Provide you with an ergonomic way of managing the complexity of your development environment.

Let’s start with a few assumptions on the required qualities of a good development environment. As much as possible, it should:

1. Remain local. Asking contributors to apt get something just to contribute to your project is both a hindrance, and a source of failure and friction in the setup process.
2. Be seamless to instantiate. Every manual step that people need to perform before working on your project is a chance to mess up, give up, or waste endless amounts of time.
3. Be defined in the repository and kept in sync with the code. It must always be possible to go back to an old version and have everything ready to work on it.
4. Be tailored to your project. Every project has different needs and there’s no “one-size fits them all” as the US air force discovered more than half a century ago.

Organist tries to make it easy for you to address these points by centralizing all the necessary configuration in a single file (or set of files) that can be checked-in to your repository, with both sane defaults and a powerful configuration mechanism.

More concretely

Given a single configuration describing your project’s whole environment (project dependencies and their configuration, the services that are required to run, etc.), Organist will drop you into an environment in which all of these are available and properly configured.

Centralizing all the configuration into a single place might look like a dangerous idea, as centralizing the mess could make it even messier if done wrong. However, Organist provides the means to avoid this trap by leveraging Nickel as its configuration language. Nickel is a language first started by Tweag, with two fundamental features that help tremendously for managing the complexity:

• Contracts provide ahead-of-time sanity checking, as well as most of the niceties of a good programming language (completion, documentation, etc.)
• The merge system makes it possible to define a (possibly complex) default configuration on the library side, and only have the client override the relevant fields.

This design draws inspiration from the NixOS module system which has shown its value in projects like NixOS and home-manager.

Nix does the heavy lifting under the hood, provisioning the dependencies (thanks to the incredible Nixpkgs software distribution), and doing the grunt work of generating and combining all the configuration files. Indeed, Nix combines a few properties that make it an awesome fit for Organist’s needs:

1. It makes it easy to build fully local and reproducible environments. This means that it’s easy to ask Nix something like “make me this very specific version of gcc available only in my current shell session”, which is incredibly valuable.
2. It is totally agnostic to what it packages. This means that we can use it indifferently for building and pulling in a 2M lines toolchain or generating a simple text file.

Organist also currently reuses the Nix command-line interface as its main entry point to avoid reimplementing the wheel (and keep it compatible with plain Nix users).

Note that Organist is not the only tool in that space. devenv in particular follows some similar goals, although with a slightly different approach. The README highlights the difference with the most prominent alternatives.

Demo time (sort-of)

Screencast of Organist in action (click for higher-res)

The repository at https://github.com/tweag/organist-example shows an example of using Organist for a real-life(-ish) project. Let’s dissect the history a bit. All the steps link to the commit that implements them for completeness (although following them isn’t necessary to get the big picture).

It all starts with a simple Python script to upload files to S3 (step 1). The script works fine, but we now want to publish it as a standalone project.

This script requires the boto3 Python library. We can pull it in with Poetry (step 2). Nothing to do with Organist so far, just plain Python development.

However, poetry itself, as well as python need to be available. We can leverage Nix for that. We’ll do it through Organist to abstract things away.

Before anything, we need a slight bit of initial boilerplate. Nix’s template mechanism can handle that for us with nix flake init -t github:nickel-lang/organist (step 3). This being done, we can edit the Organist configuration to provide us with the right environment (step 4). This means:

• Changing the base shell type from Bash to Python310. This ensures that we have a python interpreter (3.10) available, as well as some development tools to make our life easier (python-lsp-server in particular by default);

  - shells = organist.shells.Bash,
  + shells = organist.shells.Python310,

• Adding poetry to the available packages in the shell (and removing hello as it’s not very useful beyond serving as example).

  - packages = {},
  + packages.poetry = organist.import_nix "nixpkgs#poetry",

We can now enter our new development shell nix develop, and run our program (🎉). However, it requires directly accessing the Amazon S3 servers, which isn’t practical for testing. Fortunately, great third-party implementations of the S3 API exist, and we can for instance leverage minio for our tests.

This requires:

• Having the minio executable available
• Being able to run it with the right parameters
• Passing the correct parameters and credentials to our program

This is a fair amount of work to put together, especially if this has to be replicated for every contributor to our project. Leveraging a bit of Nix, a command-runner like foreman or honcho, and some manual work, we could get this all running in a reasonably automated fashion. However, Organist provides a services extension to make that easier for us and allow defining everything in a single place (step 5).

What we have to do is set services.minio to the right command (using the Nix integration to transparently fetch the minio server), and run it:

services.minio = nix-s%"
    %{organist.import_nix "nixpkgs#minio"}/bin/minio server --address :9000 ./.minio-data
  "%,

$ nix run .#start-services start
17:18:57 system  | minio.1 started (pid=461162)
...
17:18:58 minio.1 | S3-API: http://127.0.0.1:9000
...

That’s not enough, because our program doesn’t know about this local minio instance and will still try to connect to S3. But we can just set the right environment (still from our Nickel configuration) to make it look there:

env.AWS_ENDPOINT_URL = "http://localhost:9000"

And we can now run our shiny service:

$ nix develop --command poetry run python3 src/main.py

Obviously, this is just set within the scope of our development environment. Doing so doesn’t alter production in any way, so we can still use Amazon S3 (or whatever else we prefer) there.

This still isn’t ideal because the AWS_ENDPOINT_URL has to hardcode the minio port number. But this is all code! So we can just factor it out:

let minio_port = std.to_string 9031 in
{
  env.AWS_ENDPOINT_URL = "http://localhost:%{minio_port}",
  services.minio = nix-s%"
    %{organist.import_nix "nixpkgs#minio"}/bin/minio server --address :%{minio_port} ./.minio-data
  "%,
}

Going further

This got us to a working environment. But we could go further into making it a nicer environment to work with.

For instance, we can leverage Organist to generate us a good EditorConfig configuration (84bf28) or even integrate with the awesome direnv to provide automatic reloading of the environment (84696f).

Going even further

The world of development tools is an exciting one, full of marvels. Organist interfaces with a number of them, but there’s so much more that could be done!

I’m planning for instance use to let Organist generate generate GitHub actions, or integrate with tf-ncl to make it easier to deploy our project.

Don’t hesitate to contribute you own Organist module to cover any need you have that’s not already handled.

Although Organist is still in its infancy, we hope that it will make your project setup easier. We’re eager to get some feedback on it, so don’t hesitate to try it out!

It’s all happening in the Organist repository on GitHub.

In order to understand why we need nixtract, let me tell you about the “secret” value of Nixpkgs.

Is it a bird? A plane? It’s a graph!

The Nix language allows you to define the “recipe” to build anything into a package, like the sources and the steps to make the package, but also the dependencies it needs to build. For instance we can define the recipe for a package that depends on zlib.

{ stdenv, fetchFromGitHub, zlib }:

stdenv.mkDerivation {
  pname = "mypackage";
  version = "1.0.0";

  src = fetchFromGitHub {
    owner = "someowner";
    repo = "somerepo";
    rev = ...;
    hash = ...;
  };

  buildInputs = [ zlib ];
}

Such a recipe is called a “derivation”. Behind the zlib variable we use here also lies a derivation, another recipe to build said library.

Looking at this expression, we can think of it as a graph. Derivations are the nodes, and they are connected when one depends on another.

This graph reads: mypackage depends on zlib.

This is what we call a “dependency graph”.

From one to many

Once we’ve written the recipe of a package, it would be a waste not to share it, wouldn’t it? That’s what people thought, hence creating Nixpkgs, a repository of Nix expressions that define packages maintained by the Nix community.

Nixpkgs is a great endeavour, maintaining one of the largest collection of packages ever made, which we could use to draw a dependency graph!

Maintainers also add all sorts of metadata such as the name, the version, the license, the homepage, the description, and so on, which could be used to make an even richer graph.

This data is extremely valuable, and can be used for many purposes such as:

• listing all the packages that a package depends on, both direct and transitive (what we call the closure of the package)
• listing all licenses that are in the closure of a package
• if plugged into a binary cache, it would help analyze the closure size and find the dependencies that have the most impact on it
• if plugged into a CVE database, it would help flag all compromised packages

One could imagine a dashboard with all the packages used in a system, or even throughout a company’s systems, to quickly monitor, audit and act on the entire Software Supply Chain.

The information is here in Nixpkgs, but hard to use because it’s encoded in Nix expressions. nixtract’s goal is to make it accessible in a way that’s easier to consume.

extracting data from Nix expressions

Let’s consider a small Nix expression.

{
  a = 1;
  b = { c = 2; };
  d = [ { e = 3; } { f = 4; } ];
}

We can explore this nested structure like so:

$ nix eval -f example.nix --json 'a'
1
$ nix eval -f example.nix --json 'b.c'
2

'a' and 'b.c' are called attribute paths, and describe how to reach a value. This is the mechanism that is used to access packages in Nixpkgs. For example, we can install packages through their attribute path:

$ nix-env -iA pythonPackages.requests

This first accesses the pythonPackages attribute of the usual pkgs attribute set. pythonPackages is an attribute set, and we can then access its requests attribute.

One could be tempted to parse Nix expressions with a custom parser to extract the graph. Unfortunately, it gets a bit more complicated when a node has many children, represented as a list:

$ nix eval -f example.nix d --apply 'd: (builtins.elemAt d 0).e'
3

And it would quickly get too complex with Nix functions:

let
  names = ["a" "b" "c"]
  items = [0, 1, 2];
in
  builtins.listToAttrs (builtins.map (x: {name = builtins.elemAt names x; value = x;}) items)

This last expression builds a simple attribute set {a = 0; b = 1; c = 2}, but it does require to evaluate it to make any sense.

All in all, the only strategy available to extract the graph data written in Nix code is to evaluate said Nix code.

Evaluating the entire Nixpkgs

Nixpkgs being Nix code, it can be imported and evaluated, but also manipulated in Nix code!

let
  pkgs = import <nixpkgs> {};
in
  builtins.attrNames pkgs

Here pkgs is an attribute set like any in Nix, so we can manipulate it to our heart’s content.

A first approach to extract the information from Nixpkgs was thus to use a Nix expression that returns the whole graph as JSON.

Our more regular readers probably have heard about this on this very same blog about a year ago. However, at this time, we built the graph of packages that are top-level, and only them. As it turns out, this is not sufficient for a complete analysis; what we need is the entire graph of derivations, even the ones not accessible at the top-level.

So now, the idea is to recursively go down pkgs, find derivations, transform them into a JSON-like attribute set, but also keep recursing into its dependencies.

For the curious, below is a small snippet demonstrating that idea. Don’t worry if it looks intimidating, you don’t need to understand it to read this blog post.

{ nixpkgs, system }:
let
  # recurse :: string -> any -> attrs | null
  recurse =
    name: value:
      if !(nixpkgs.lib.isDerivation value) then
        # recurse when possible
        if (value.recurseForDerivations or false)
        then
          builtins.mapAttrs recurse value
        else
          null
      # process derivation and recurse into inputs
      else
        {
          # path to the evaluated derivation file
          derivationPath = value.drvPath;
          # one could add any information

          # recurse into inputs
          buildInputs = map (recurse null) (value.buildInputs or [ ]);
          # do the same for other inputs...
        }
    ;
in
  # build the value that will be yielded
  map
    # go through a bit of processing to not have the children nodes written in the node
    (drv: {
      inherit (drv) derivationPath;
      buildInputs = map (inputDrv: inputDrv.derivationPath) (drv.buildInputs or []);
    })
    # process recursively
    (nixpkgs.lib.collect
      (x: (x.derivationPath or null) != null)
      (builtins.mapAttrs recurse nixpkgs.legacyPackages.${system})
    )

If you would like to learn more about it, you can check out the complete expression we used: nixpkgs-graph-explorer/core/extract/nixpkgs-graph.nix¹. It handles the many tricks of evaluating Nixpkgs.

The idea would then be to evaluate it and get the output data.

$ nix eval -f nixpkgs-to-graph.nix --json

Unfortunately, that dream quickly came to an end when recursing over the entire Nixpkgs. As much as we tried to tailor an expression to recursively describe, Nix kept filling the RAM and made our laptops crash.

But if one Nix can’t do it all, maybe many Nix-s can!

Towards nixtract

nixtract uses not one, but two Nix expression.

The first one focuses on finding all the top-level derivations that can be directly accessed from pkgs. It differs from the snippet shown earlier in that

• It does not recurse on the inputs of the derivation,
• It outputs nothing more than the attribute path and the output path,
• The output is streamed rather than batched.

$ TARGET_FLAKE_REF="nixpkgs" TARGET_SYSTEM="x86_64-linux" nix eval --json --file ./nixtract/find-attribute-paths.nix
trace: {"foundDrvs":[{"attributePath":"AMB-plugins.out","derivationPath":"/nix/store/p060rxjqna1pnmg63yz6j7ya1as1wk4k-AMB-plugins-0.8.1.drv","outputPath":"/nix/store/z15irhv9lbr3f86fgifzbhrnga2m42ji-AMB-plugins-0.8.1"}]}
trace: {"foundDrvs":[{"attributePath":"ArchiSteamFarm.out","derivationPath":"/nix/store/s57c44cqwn7sw7vc7b7zx0na4snvkh9c-archisteamfarm-5.4.4.5.drv","outputPath":"/nix/store/ka2wsngqkblcbl3awbcmhlpxllrddif4-archisteamfarm-5.4.4.5"}]}
...

The second one focuses on following an attribute path and “describing” the derivation.

$ TARGET_FLAKE_REF="nixpkgs" TARGET_SYSTEM="x86_64-linux" TARGET_ATTRIBUTE_PATH="hello" nix eval --json --file ./nixtract/describe-derivation.nix
{"attributePath":"hello","buildInputs":[],"derivationPath":"/nix/store/r91xxmyayj90xlr0378r817ly09khpxg-hello-2.12.1.drv","name":"hello-2.12.1","nixpkgsMetadata":{"broken":false,"license":"GNU General Public License v3.0 or later","pname":"hello","version":"2.12.1"},"outputPath":"/nix/store/7syg3lif5ik17dsrwgk3s00s116q87by-hello-2.12.1","outputs":[{"name":"out","outputPath":"/nix/store/7syg3lif5ik17dsrwgk3s00s116q87by-hello-2.12.1"}],"parsedName":{"name":"hello","version":"2.12.1"}

In Python, we make a queue of attribute paths that we need to “describe”, fed by the first expression ran in a process, and we process each of them using the second expression ran in other processes.

But the first expression is not the only one contributing to the queue. When the second expression, the one that “describes”, visits a derivation that has inputs, these input derivations will be added by the Python script to the queue if they haven’t already been added. This ensures that not only top level derivations are described, but all the derivations in the closure of each visited derivation are as well.

Instead of recursing everything at once in one Nix process that ends up gobbling up all our RAM, we spawn many small Nix processes that don’t explode. An extra benefit is that we can now parallelize the extraction and get the data faster.

and voilà!

All in all, you only use a single command line to start the magic:

$ nixtract -
{"attribute_path": "AMB-plugins.out", "derivation_path": "/nix/store/90bclfvqnc0mzn6vsd99bzl383vl2j04-AMB-plugins-0.8.1.drv", "output_path": "/nix/store/7n05ir19xx0086dk1wq1zm1gq8g1ymyr-AMB-plugins-0.8.1", "outputs": [{"name": "out", "output_path": "/nix/store/7n05ir19xx0086dk1wq1zm1gq8g1ymyr-AMB-plugins-0.8.1"}], "name": "AMB-plugins-0.8.1", "parsed_name": {"name": "AMB-plugins", "version": "0.8.1"}, "nixpkgs_metadata": {"pname": "AMB-plugins", "version": "0.8.1", "broken": false, "license": "GNU General Public License v2.0 or later"}, "build_inputs": [{"attribute_path": "AMB-plugins.out.buildInputs.0", "build_input_type": "build_input", "output_path": "/nix/store/ib5bh4b9fl40zamlx0zpfgw8ygaj01l5-ladspa.h-1.15"}]}
...

{"attribute_path": "CHOWTapeModel.out.buildInputs.2", "derivation_path": "/nix/store/pvdzfr30s1xc8qcvavmmkj1j51lpnifp-curl-8.3.0.drv", "output_path": "/nix/store/r4638iv5a646jz14rl02r6r9cqhggaky-curl-8.3.0-dev", "outputs": [{"name": "bin", "output_path": "/nix/store/yiip8b2ks99ixn5jf9jlx0bsxl5r2h2m-curl-8.3.0-bin"}, {"name": "dev", "output_path": "/nix/store/r4638iv5a646jz14rl02r6r9cqhggaky-curl-8.3.0-dev"}, {"name": "out", "output_path": "/nix/store/bhmynyjwzc2r6iqf7fhc3yzjcv3paiwa-curl-8.3.0"}, {"name": "man", "output_path": "/nix/store/c19b760rmwv0j57bgscxmic9qkylihzg-curl-8.3.0-man"}, {"name": "devdoc", "output_path": "/nix/store/ad15cgxydfspm6sqda4plrv2vf2kfib9-curl-8.3.0-devdoc"}, {"name": "debug", "output_path": "/nix/store/bv42bixmsj38cdfdrf08760pxc5z03pq-curl-8.3.0-debug"}], "name": "curl-8.3.0", "parsed_name": {"name": "curl", "version": "8.3.0"}, "nixpkgs_metadata": {"pname": "curl", "version": "8.3.0", "broken": false, "license": "curl License"}, "build_inputs": [{"attribute_path": "CHOWTapeModel.out.buildInputs.2.nativeBuildInputs.0", "build_input_type": "native_build_input", "output_path": "/nix/store/5daca24rn22c65ff25lc6z0g0imfphvr-pkg-config-wrapper-0.29.2"}, {"attribute_path": "CHOWTapeModel.out.buildInputs.2.nativeBuildInputs.1", "build_input_type": "native_build_input", "output_path": "/nix/store/2j7b1ngdvqd0bidb6bn9icskwm6sq63v-perl-5.38.0"}, {"attribute_path": "CHOWTapeModel.out.buildInputs.2.propagatedBuildInputs.0", "build_input_type": "propagated_build_input", "output_path": "/nix/store/xz08admjq3viprgxnb6gc9xvbbhdxq7h-brotli-1.1.0-dev"}, {"attribute_path": "CHOWTapeModel.out.buildInputs.2.propagatedBuildInputs.1", "build_input_type": "propagated_build_input", "output_path": "/nix/store/s64na02z7fd3q6d04br9ivbdrqcmx5vj-libkrb5-1.20.1-dev"}, {"attribute_path": "CHOWTapeModel.out.buildInputs.2.propagatedBuildInputs.2", "build_input_type": "propagated_build_input", "output_path": "/nix/store/46dnqv0sjm5g1sgd4r0rqp50sv7wr1ma-nghttp2-1.54.0-dev"}, {"attribute_path": "CHOWTapeModel.out.buildInputs.2.propagatedBuildInputs.3", "build_input_type": "propagated_build_input", "output_path": "/nix/store/84d89lbdfdzy1wb363k3avl8qb8i6lcb-libidn2-2.3.4-dev"}, {"attribute_path": "CHOWTapeModel.out.buildInputs.2.propagatedBuildInputs.4", "build_input_type": "propagated_build_input", "output_path": "/nix/store/lqsfdc4p9vax2h55csza9iw46zjpghl6-openssl-3.0.10-dev"}, {"attribute_path": "CHOWTapeModel.out.buildInputs.2.propagatedBuildInputs.5", "build_input_type": "propagated_build_input", "output_path": "/nix/store/ifvv1iq1fq7j3z7ykz997gnba8a2yy5m-libssh2-1.11.0-dev"}, {"attribute_path": "CHOWTapeModel.out.buildInputs.2.propagatedBuildInputs.6", "build_input_type": "propagated_build_input", "output_path": "/nix/store/9m8fx7phv5gr67b2yd5a43p287hg10g0-zlib-1.3-dev"}, {"attribute_path": "CHOWTapeModel.out.buildInputs.2.propagatedBuildInputs.7", "build_input_type": "propagated_build_input", "output_path": "/nix/store/zgflsh0x6c8lka6y0n2diiph6ahiglxh-zstd-1.5.5-dev"}]}
...

This command outputs the information about all derivations, top-level or any dependency, from a flake (by default nixpkgs from your flake registry).

What’s more, our processing is threaded in order to stream the result, to make sure that the user does not have to wait for everything to be processed before being able to start consuming the data. This is really convenient to pipe with other tools like jq:

$ nixtract - | jq -r '.derivation_path'
/nix/store/90bclfvqnc0mzn6vsd99bzl383vl2j04-AMB-plugins-0.8.1.drv
/nix/store/vwz2w3arkprzxc28d0f621hbv8gq468f-ArchiSteamFarm-5.4.9.3.drv
...

Not only can you extract the data from the nixpkgs flake for your current system, but you can extract the data from any flake and for any system:

$ nixtract --target-flake-ref 'github:nixos/nixpkgs/23.05' --target-system 'x86_64-linux' -
{"attribute_path": "AMB-plugins.out", "derivation_path": "/nix/store/ddw13azl05jp816s57zzkqlf7qa2c7xg-AMB-plugins-0.8.1.drv", "output_path": "/nix/store/sbkj5l8ynhywr1fqszxr91g043gvxjc1-AMB-plugins-0.8.1", "outputs": [{"name": "out", "output_path": "/nix/store/sbkj5l8ynhywr1fqszxr91g043gvxjc1-AMB-plugins-0.8.1"}], "name": "AMB-plugins-0.8.1", "parsed_name": {"name": "AMB-plugins", "version": "0.8.1"}, "nixpkgs_metadata": {"pname": "AMB-plugins", "version": "0.8.1", "broken": false, "license": "GNU General Public License v2.0 or later"}, "build_inputs": [{"attribute_path": "AMB-plugins.out.buildInputs.0", "build_input_type": "build_input", "output_path": "/nix/store/wkxygwfn22g8fjs6alq9j1x2jiaw679k-ladspa.h-1.15"}]}
{"attribute_path": "AMB-plugins.out.buildInputs.0", "derivation_path": "/nix/store/3j3lzndm8ky97hc59s3bfdf6ln38wg2h-ladspa.h-1.15.drv", "output_path": "/nix/store/wkxygwfn22g8fjs6alq9j1x2jiaw679k-ladspa.h-1.15", "outputs": [{"name": "out", "output_path": "/nix/store/wkxygwfn22g8fjs6alq9j1x2jiaw679k-ladspa.h-1.15"}], "name": "ladspa.h-1.15", "parsed_name": {"name": "ladspa.h", "version": "1.15"}, "nixpkgs_metadata": {"pname": "ladspa.h", "version": "1.15", "broken": false, "license": "GNU Library General Public License v2"}, "build_inputs": []}
...

This is just the beginning

We have released nixtract 0.1.0, check out the README and try it now!

The data extracted by nixtract is somewhat basic, but you can expect it to extract more metadata from derivations in the future.

nixtract focuses on extracting the data, which is hardly enough. We hope to combine nixtract with other tools so that anyone is able to scan the whole Nixpkgs graph as they need. It would also be exciting to combine this source of data with others, like binary caches and CVE databases, to enable even more use cases.

We hope that nixtract becomes a foundational brick that sparks new efforts towards better understanding Nixpkgs and improving Software Supply Chain management.

A bit more than one year ago, we released the very first public version Nickel (0.1). Throughout various write-ups and public talks (1, 2, 3), we’ve been telling the story of our dissatisfaction with the state of configuration management.

The need for a New Deal

Configuration is everywhere. The manifest of a web app, the configuration of an Apache virtual host, an Infrastructure-as-Code (IaC) cloud deployment (Terraform, Kubernetes, etc.).

Configuration is more often than not considered a second-class engineering discipline, like a side activity of software engineering. That’s a dangerous mistake: with the advent of IaC for the cloud, configuration has become an important aspect of modern software systems, and a critical point of failure.

All the different but connected configurations composing a system are scattered across many languages, tools, and services (JSON, YAML, HCL, Puppet, Apache’s Tcl, and so on). Configuration management is indeed a cross-cutting concern, making failures harder to predict and often spectacular.

In the last decade, studies have shown that misconfigurations were the second largest cause of service-level disruptions in one of Google’s main production services ¹. Misconfigurations also contribute to 16% of production incidents at Facebook ², including the worst-ever outage of Facebook and Instagram that occurred in March 2019 ⁴. Fastly’s³ outage on the 8th June 2021, which basically broke a substantial part of the internet, was triggered by a configuration issue.

Modern configurations are complex. They require new tools to be dealt with, which is why we developed the Nickel configuration language.

Nickel 1.0

Nickel is a lightweight and generic configuration language. It can replace YAML as your new application’s configuration language, or it can generate static configuration files (YAML, JSON or TOML) to be fed to existing tools.

Unlike YAML, though, it anticipates large configurations by being programmable and modular. To minimize the risk of misconfigurations, Nickel features (opt-in) static typing and contracts, a powerful and extensible data validation framework.

Since the initial release (0.1), we’ve refined the semantics enough to be confident in a core design that is unlikely to radically change.

Since the previous stable version (0.3.1), efforts have been made on three principal fronts: tooling (in particular the language server), the core language semantics (contracts, metadata, and merging), and the surface language (the syntax and the stdlib). Please see the release notes for more details.

Tooling & set-up

Follow the getting started guide from Nickel’s website to get a working binary. Nickel also comes with:

• An LSP language server.
• A REPL nickel repl, a markdown documentation generator nickel doc and a nickel query command to retrieve metadata, types and contracts from code.
• Plugins for (Neo)Vim, VSCode, Emacs, and a tree-sitter grammar.
• A code formatter, thanks to Tweag’s tree-sitter-based Topiary.

Watch the companion video for a tour of these tools and features.

A primer on Nickel

Let me walk you through a small example showing what you can do in Nickel 1.0. You can try the code examples from this section in the online Nickel playground.

Just a fancy JSON

I’ll use a basic Kubernetes deployment of a MySQL service as a working example. The following is a direct conversion of a good chunk of mysql.yaml into Nickel syntax, omitting uninteresting values:

{
  apiVersion = "v1",
  kind = "Pod",
  metadata = {
    name = "mysql",
    labels.name = "mysql",
  },

  spec = {
    containers = [
      {
        resources = {
          image = "mysql",
          name = "mysql",
          ports = [
            {
              containerPort = 3306,
              name = "mysql",
            }
          ],
          volumeMounts = [
            {
              # name must match the volume name below
              name = "mysql-persistent-storage",
              # mount path within the container
              mountPath = "/var/lib/mysql",
            }
          ],
        }
      }
    ],
    volumes = [
      {
        name = "mysql-persistent-storage",
        cinder = {
          volumeID = "bd82f7e2-wece-4c01-a505-4acf60b07f4a",
          fsType = "ext4",
        },
      }
    ]
  }
}

This snippet looks like JSON, with a few minor syntax differences. Nickel has indeed the same primitive data types as JSON: numbers (arbitrary precision rationals), strings, arrays, and records (objects in JSON).

The previous example has a lot of repetition: the string "mysql", the name of the app, occurs several times.

Besides, a comment mentions that the name inside the volumeMounts field must match the name of the volume defined inside volumes. Developers are responsible for maintaining this invariant by hand. The absence of a single source of truth might lead to inconsistencies.

Finally, imagine that you now need to reuse the previous configuration several times with slight variations, where the app name and the MySQL port number may change.

This can’t be solved in pure YAML: all you can do is to copy and paste data, and try to manually ensure that copies always all agree. Unlike YAML though, Nickel is programmable.

Reusable configuration

Let’s upgrade our very first example:

# mysql-module.ncl
{
  config | not_exported = {
    port,
    app_name,
    volume_name = "mysql-persistent-storage",
  },

  apiVersion = "v1",
  kind = "Pod",
  metadata = {
    name = config.app_name,
    labels.name = config.app_name,
  },

  spec = {
    containers = [
      {
        resources = {
          limits.cpu = 0.5,
          image = "mysql",
          name = config.app_name,
          ports = [
            {
              containerPort = config.port,
              name = "mysql",
            }
          ],
          volumeMounts = [
            {
              name = config.volume_name,
              # mount path within the container
              mountPath = "/var/lib/mysql",
            }
          ],
        }
      }
    ],
    volumes = [
      {
        name = config.volume_name,
        cinder = {
          volumeID = "bd82f7e2-wece-4c01-a505-4acf60b07f4a",
          fsType = "ext4",
        },
      }
    ]
  }
}

The most important change is the new config field. The rest is almost the same as before, but I replaced the strings "mysql" with config.app_name, "mysql-persistent-storage" with config.volume_name and the hard-coded port number 3306 with config.port.

We can directly use the new fields config.xxx from within other fields. Indeed, in Nickel, you can refer to another part of a configuration you are defining from inside the very same configuration. It’s a natural way to describe data dependencies, when parts of the configuration are generated from other parts.

The | symbol attaches metadata to fields. Here, | not_exported indicates that config is an internal value that shouldn’t appear in the final exported YAML.

This explains most of the new machinery. But this configuration has still something off. Let us try to export it:

$ nickel export -f mysql-module.ncl --format yaml
error: missing definition for `port`
   ┌─ mysql.ncl:33:36
   │
 2 │     config | not_exported = {
   │ ╭───────────────────────────'
 3 │ │     port,
 4 │ │     app_name,
 5 │ │     volume_name = "mysql-persistent-storage",
 6 │ │   },
   │ ╰───' in this record
   · │
33 │               containerPort = config.port,
   │                               -------^^^^
   │                               │      │
   │                               │      required here
   │                               accessed here

Indeed, port and app_name don’t have a value! In fact, you should view this snippet as a partial configuration, a configuration with holes to be filled.

To recover a complete configuration, we need the merge operator &. Merging is a primitive operation that recursively combines records together. Merge is also able to provide a definite value to the fields app_name and port:

# file: mysql-final.ncl
(import "mysql-module.ncl") & {
  config = {
    app_name = "mysql-backend",
    port = 10500,
  }
}

Running nickel export --format yaml -f mysql-final.ncl will now produce a valid YAML configuration.

Partial configurations bear similarities with functions: both are a solution to the problem of how to make repetitive code reusable and composable. But the former seems more adapted to writing reusable configuration snippets.

It’s trivial to assemble several partial configurations together (as long as there’s no conflict): just merge them together.

Partial configurations are data, which can be queried, inspected, and transformed, as long as the missing fields aren’t required. For example, you can get the list of fields of our partial configuration:

$ nickel query -f mysql-module.ncl

Available fields
• apiVersion
• config
• kind
• metadata
• spec

Finally, partial configurations are naturally overridable. Recall that mysql-final.ncl contains the final complete configuration, and consider this example:

# file: override.ncl
(import "mysql-final.ncl") & {
  config.app_name | force = "mysql_overridden",
  metadata.labels.overridden = "true",
}

Exporting override.ncl produces a configuration where all the values depending on config.app_name (directly or indirectly) are updated to use the new value "mysql_overridden", and where metadata.labels has a new entry overridden set to "true".

Overriding is useful to tweak existing code that you don’t control, and wasn’t written to be customizable in the first place. It’s probably better to do without whenever possible, but for some application domains, it simply can’t be avoided.

Partial configurations are automatically extensible: you don’t have to even think about it.

On the other hand, functions are opaque values, which need to be fed with arguments before you can do anything with them. Assembling many of them, inspecting them before they are applied or making their result overridable range from technically possible but much more cumbersome to downright impossible.

Correct configurations

Nickel has two main mechanisms to ensure correctness and prevent misconfigurations as much as possible: static typing and contracts.

Static typing is particularly adapted for small reusable functions. The typechecker is rigorous and catches errors early, before the code path is even triggered.

For configuration data, we tend to use contracts. Contracts are a principled way of writing and applying runtime data validators. It feels like typing when used, relying on annotations and contract constructors (array contract, function contract, etc.), but the checks are performed at runtime. In return, you can easily define your own contracts and compose them with the existing ones.

Contracts are useful to validate the produced configuration (in that case, they will probably come from a library or be automatically generated from external schemas such as when writing Terraform configurations in Nickel). They can validate inputs as well:

# file: mysql-module-safe.ncl
let StartsWith = fun prefix =>
  std.contract.from_predicate (std.string.is_match "^%{prefix}")
in

{
  config | not_exported = {
    app_name
      | String
      | StartsWith "mysql"
      | doc m%"
          The name of the mysql application. The name must start with `"mysql"`,
          as enforced by the `StartsWith` contract.
        "%
      | default
      = "mysql",
    port
      | Number
      | default
      = 3306,
    volume_name = "mysql-persistent-storage",
  },

  # ...
}

The builtin String contract and the custom contract StartsWith "mysql" have been attached to app_name. The latter enforces that the value starts with "mysql", if we have this requirement for some reason. We won’t enter the details of custom contracts here, but it’s a pretty straightforward function.

We’ve used other metadata as well: doc is for documentation, while default indicates that the following definition is a default value. Metadata are leveraged by the Nickel tooling (the LSP, nickel query, nickel doc, etc.)

If we provide a faulty value for app_name, the evaluation will raise a proper error.

Going further

You can look at the main README for a general description of the project. The first blog post series explains the inception of Nickel, and the following posts focus on specific aspects of the language. The most complete source remains the user manual.

Configure your configuration

Zooming out from technical details, I would like to paint a broader picture here. My overall impression is that using bare YAML for e.g. Kubernetes is like programming in assembly. It’s certainly doable, but not desirable. It’s tedious, low-level, and lacks the minimal abstractions to make it scalable.

Some solutions do make configuration more flexible: YAML templating (Helm), tool-specific configuration languages (HCL), etc. But many of them feel like a band-aid over pure JSON or YAML which somehow accidentally grew up to become semi-programming languages.

Our final example is a snippet mapping a pair of high-level values — the only values we care configuring, app_name and port — to a “low-level” Kubernetes configuration. Once written, the only remaining thing to do is to… configure your configuration!

(import "mysql-module.ncl") & {
  config.app_name = "mysql_backup",
  config.port_number = 10400
}

Nickel allows to provide a well-defined interface for reusable parts of configuration, while still providing an escape hatch to override anything else when you need it. Such configuration snippets can be reused, composed, and validated thanks to types and contracts. Although Nickel undeniably brings in complexity, Nickel might paradoxically empower users to make configuration simple again.

If JSON is enough for your use-case, that’s perfect! There’s really no better place to be. But if you’ve ever felt underequipped to handle, write and evolve large configurations, Nickel might be for you.

Conclusion

We are happy to announce the 1.0 milestone for the Nickel configuration language. You can use it wherever you would normally use JSON, YAML, or TOML, but feel limited by using static text or ad-hoc templating languages. You can use it if your configuration is spread around many tools and ad-hoc configuration languages. Nickel could become one fit-them-all configuration language, enabling the sharing of the abstractions, code, schemas (contracts) and tooling across all your stack. Don’t hesitate to check the companion video to help you getting set up.

In our next blog post, we’ll show how to configure Terraform providers using Nickel, and thereby gain the ability to check the code against provider-specific contracts, ahead of performing the deployments. Stay tuned!

Your feedback, ideas, and opinions are invaluable: please use Nickel, break it, do cool things we haven’t even imagined, and most importantly, please let us know about it! Email us at hello@tweag.io or go to Nickel’s open source GitHub repository.

Our involvement didn’t end there. At FOSDEM:

• Théophane presented the new Nix (core) team.
• Guillaume Desforges lightened the mood with his enthusiastic talk on making anyone use Nix.
• I went on to introduce Nixel, a framework to write Nix expressions in Nickel.

Followed right away by CfgMgmtCamp:

• Bryan first did a lightning talk to tease Nix, then took curious people on a deeper dive during a 50-minute gentle introduction to Nix.
• Viktor Kleen presented his work on Terraform-Nickel, a library to use Nickel in place of HCL for Terraform.
• I talked about how to write modular and customizable configuration using Nickel’s merging system.

What follows are my takeaways on the trends and the trajectory of infrastructure and configuration management. Please don’t take my word as a confident prediction, as predictions don’t age well. This post is merely a patchwork of my biased, fresh impressions — as a programming language nerd rabbit-holed into devops and infrastructure by Nix — on my way back home.

Self-service infrastructure

Several key speakers at CfgMgmtCamp emphasized how much infrastructure is critical for many business services and how, more often than not, it is a bottleneck in practice.

Developers must be able to spin up new services rapidly, not in hours or days. They can’t wait one week for someone from the infra team to finally find time, between daily maintenance tasks, to hit the green button (or do something more complex), potentially followed by a long painful back-and-forth. Non-technical roles, such as marketing and sales, might also need to get a small website or a service up, but are even less empowered.

This doesn’t mean one should give everyone full access to the infrastructure. I’ve rather heard a call for simpler, safer, and better automated processes. For example, having a simple web interface for spinning up instances, with a clear policy on cleaning inactive resources. The interface shouldn’t require much infrastructure knowledge nor expose low-level options: this isn’t simply about granting people access to an AWS console. If users are developers, this can be achieved using infrastructure as code as well, with adapted restrictions.

Continuous integration, continuous delivery, one-off experiments — all those components should be automated, fully integrated in the software development workflow, and just a few clicks (or keystrokes, for you Vimmers) away.

YAML is dead

I’m sorry, that was just clickbait. YAML has been pronounced dead for the last 10 years or so, and unfortunately, here it stands.

But something is in the air. There is an undeniable movement toward a unified and structured approach to writing and managing configuration.

Scattering configuration data, schemas and knowledge across many different tools, written in many different languages (HCL, YAML, JSON, TOML, Puppet, Ansible, Helm, etc.) isn’t sustainable. And I feel like we can state a variant of Greenspuns’ tenth rule of programming for configuration:

Every sufficiently complicated infrastructure tool’s native configuration language is an ad-hoc, informally-specified, bug-ridden, arbitrarily-limited reinvention of half of CommonLisp.

This is exactly why we created Nickel. Projects that were the new kids on the block a few years ago — I’m thinking about Pulumi or CUE, for example — are gaining strength and run today in production. Those are not curiosities any more. The actors behind those tools seem to share the same dissatisfaction with the state of configuration management and infrastructure engineering. While our experience and vision led us to come up with our own answer (Nickel), we are all trying to unify configuration management by using a common language or platform.

Declarative, but what about interactive?

Adam Jacob’s opening talk on Breaking The Rules, as well as Ringo De Smet’s talk on Pulumi’s imperative on top of declarative approach made me think about the user experience aspect of declarative configuration. I’m still convinced that the declarative approach is ultimately the right one, or the only sustainable one to this day.

However, there is something to be said about offering different interfaces to this declarative layer: Pulumi is one example, providing an imperative API to manipulate it. npm add is another simple example of an imperative interface to a declarative configuration (packages.json).

Adam introduced the notion of “infrastructure as a model”, instead of code, to which code would only be one possible interface. Which reminded me of similar internal discussion at Tweag about projectional editors, which is in fact the very same idea but applied to programming in general. Textual code is just one view, but other views — such as a graphical interface — of the abstract representation of a program are also possible, and more desirable for some tasks (refactoring, renaming, diffing, visualizing, …). Typically, a diagrammatic view is often better than text at clarifying the relations and dependencies between the entities being managed (e.g. Terraform resources). To each task, its projection?

Nix is getting stronger every year

Ron Efroni, from Flox, often makes the half-joke that the number of GitHub stars on the Nix repos are following Moore’s law: they double every two years. So far, he’s mostly right.

I was impressed by the number of people flooding the Nix DevRoom at FOSDEM (the pictures at the beginning of this post), queuing in the corridor to wait for a free spot.

At CfgMgmtCamp, Bryan’s lightning talk on Nix took place on the main track during the very first morning. The (huge) auditorium was full, fitting several hundred people. When he asked who knew about Nix, I expected a solid 50 people to raise their hand. More than half, probably around two third of the audience, raised their hand!

Nix is vibrant

A lot of cool things are happening in the Nix world. The ecosystem is evolving at a fast pace. The NixOS Foundation, itself rather new, is structuring to handle the change of scale, for example with a dedicated Nix maintenance team. Flox just released the first version of their product. Domen Kozar introduced devenv.sh, which has gained traction rapidly. We presented Nixel, the starting point of a whole new user experience for writing Nix.

Nickel

We got a lot of encouraging words after our various talks on Nickel. The project has been making steady progress in the long run, but the hard and technical work on the core language, the type system and the semantics of merging is naturally less palpable.

However, we have reached the point from where we can start (and have started, in fact) to harvest impactful outcomes, which is exciting. A handful of months’ work on the LSP by our own Gaga Ebresafe and we are already able to provide completions based on definitions, types and contracts, in-code documentation, code navigation and typechecking live in the editor, which is quite ahead of the current Nix experience. Although it’s only the beginning, we can already write basic derivations and modular and overridable development shells using Nixel. Daniele Palombi is doing an internship to bring incremental evaluation and self-adjusting computations to the interpreter, for fast re-evaluation of a configuration upon a small change.

Concluding words

It’s hard to say where the current momentum will bring us. Getting people excited and aware of a cool technology is not the same as making it established and used in production. There are many non-technical determining factors, even irrational ones. Nonetheless, it’s clear that infrastructure engineering is distinguished by the ebullience of its youth. It’s an exciting time to be around!

Like many other language-specific package managers (e.g. cargo, cabal, etc.), opam performs four main tasks:

1. Download the sources.
2. Resolve the needed dependencies of a package.
3. Provide those dependencies such that the build system can find them.
4. Run the build system.

It is pretty good at this. However, there are some problems with step (4):

• The build is not properly isolated: it can (in theory) fetch arbitrary things from the network, access arbitrary files on the filesystem, and even modify other packages. This allows for irreproducible builds, and makes it easy to forget to explicitly list a system dependency if it happens to be installed on the author’s system.
• “External” (system) dependencies, such as non-OCaml binaries and libraries, are taken from the user’s distribution repository (using the distribution’s package manager, e.g. apt-get), resulting in version inconsistencies and the possibility for breakage.
• The builds are not easily cached and reused, meaning you have to compile packages locally.

Also, opam’s user interface is based around imperative commands, meaning that the developer environment setup has to be a script that modifies a switch (opam’s term for an independent collection of interdependent packages), which is fragile, difficult to update, and prone to inconsistencies. While there are tools that solve some of those issues, this can still be painful.

Finally, opam is a package manager for OCaml. It can’t easily integrate with other programming languages. This is a problem for modern software stacks, which often feature multiple programming languages in a single project.

Introducing opam-nix

If you’re familiar with Nix, you might have noticed that it doesn’t have any of the aforementioned problems. However, Nix on its own doesn’t know how to build opam packages. The solution to this? Make a library that “translates” opam packages into a format which Nix can understand. That’s exactly what opam-nix is!

opam-nix provides low-level functions which parse opam files into Nix data structures (using opam-file-format), interpret those data structures, resolve the dependency tree (using opam itself), and turn the dependency tree into derivations. It also has some overrides which ensure that a lot of popular packages build and function correctly.

If that sounds scary, don’t worry, opam-nix also provides high-level tools which make Nixifying your OCaml projects a breeze. In this blog post, we’ll show some examples of how to quickly Nixify your existing opam-based projects. Whether you’re working on improving the onboarding experience on a project or are an OCaml developer yourself, we hope you’ll find it useful.

In the following, we assume that you already have Nix installed and are using flakes. Also, the templates are suited to building opam projects.

Simple package

To get started, fetch the template provided with opam-nix. From your project’s root:

$ nix flake init -t github:tweag/opam-nix
$ git add flake.nix

Nix will create a flake.nix for you. Note that you have to add it to the Git index, otherwise Nix will not pick it up. Open it with your editor, look through the file, and replace the throw with your package name, as specified in the comment:

   outputs = { self, flake-utils, opam-nix, nixpkgs }@inputs:
-    # Don't forget to put the package name instead of `throw':
-    let package = throw "Put the package name here!";
+    let package = "my-package";
     in flake-utils.lib.eachDefaultSystem (system:

Nix is famous for its “shells”: ad-hoc, on-the-fly environments, allowing developers to quickly get started on and switch between projects. opam-nix allows you to leverage this potential to make onboarding to your projects quick and easy.

Now run

$ nix develop

Nix will download and lock opam-nix, nixpkgs, opam-repository, and some other dependencies, then build all the dependencies of your project. Once that’s done, it will drop you into a shell with all the dependencies available.

Unfortunately, that won’t always happen; opam-nix can’t provide perfect compatibility with opam. Most errors you will get are actually symptoms of problematic packaging of your dependencies, e.g. missing a system dependency requirement, arbitrary network access during the build, etc. The proper way to fix such errors is to fix the packaging upstream. However, you can also just override the dependency using the overlay in your flake.nix. You can read more about overlays and package overrides if you are not familiar. For example, you can change the commands used to build a package like this:

         overlay = final: prev:
           {
             # Your overrides go here
+            dune = prev.dune.overrideAttrs (_: { buildPhase = "make release"; });
           };
       in {
         legacyPackages = scope.overrideScope' overlay;

Additionally, if your project requires libraries or tools written in other languages, you can look for them in nixpkgs or package them with Nix (maybe using other *-nix tools). Once you have the package, you can simply then inject it into buildInputs of your project using overrideAttrs inside an overlay.

Once you get to the shell, you can use the usual tools to build your package. Typically, that would be something like dune build or make. You can also use nixpkgs phases to build and test your project using commands specified in the opam file:

$ eval "$prePatch"
$ eval "$configurePhase"
$ eval "$buildPhase"
$ eval "$checkPhase"

Note that this approach combines the benefits of Nix (reproducibility, reliable caching, isolation) for your dependencies with the benefits of your build system (fast, incremental builds, granular caching) for your project itself.

If you want to go all-in on Nix, you can build your project as a Nix derivation too. Just run:

$ nix build

This will build and check your project. You can find the build artifacts in the result folder.

Fully-featured development environment

Many developers are not content with simply being able to build the package, though; they also want to have modern amenities such as a language server to get interactive documentation, type information, and code navigation.

Worry not! opam-nix can also help with that. We’ll use a different template here, so make sure to back up any changes to flake.nix you might have made in the previous section.

$ mv flake.nix flake.nix.bkp
$ nix flake init -t github:tweag/opam-nix#multi-package
$ git add flake.nix

Replicate the overrides you made before, if any. You don’t need to specify the package name here, since this template picks up all packages in your repository. It’s also a good idea to look through flake.nix, as it might give you ideas for improving your development experience.

You can now use

nix develop

to get a development environment, as before.

However, now you also get ocaml-lsp-server and ocamlformat available. You can add other OCaml tools to the devPackagesQuery as well. For your editor to pick them up, you’ll have to start it from this environment. Alternatively, since this template provides an .envrc, you can use direnv, which has integrations with many popular editors.

Note that for ocaml-lsp to work, it must be able to find type information for your project. Typically, you can ensure this by running

$ dune build @check

If you wish, you can also build your package with:

$ nix build .#<your-package>

Diving deeper

Since opam-nix is a Nix tool, you get a lot of advantages of Nix, like reproducibility, easy CI with binary caching for even faster developer onboarding, integration with NixOS to get reproducible system deployments, or with Docker for compatibility with most of the world.

Also, because opam-nix follows the philosophy of composing small, low-level parts to make a bigger, user-friendly whole, you can make use of it even if your project doesn’t fit the requirements of buildOpamProject.

For dune-project-based projects, you can use buildDuneProject instead of buildOpamProject.

If you just have a single opam export, that’s then imported with opam import. This setup can be replicated with a snippet like this:

let
  # This is a list of package names installed in the switch
  switch = opam-nix.opamListToQuery (opam-nix.fromOPAM ./opam.export).installed;

  scope = with opam-nix;
    queryToScope { repos = [ opamRepository (makeOpamRepo ./.) ]; }
    (switch // { my-package = "dev"; });
in scope.my-package

Or, if you want to build a Mirage unikernel, you can do so using hillingar, an opam-nix- based tool.

If you want to speed up your project by avoiding Import From Derivation, opam-nix supports haskell.nix-style materialization.

Also, if you’re using jupyenv’s OCaml kernel, you’re actually using opam-nix under the hood; this means all the tricks mentioned previously can also work there

You can check out the documentation for all the public functions provided by opam-nix in the README. Let us know if you make something cool with this!

Inspirations & alternatives

opam-nix wouldn’t be possible without opam and opam-file-format. It uses them directly and reimplements some parts of them in Nix.

The way opam-nix works is similar to crate2nix, cabal2nix, and poetry2nix. Inspiration for many technical decisions was drawn from these projects.

Finally, there are a couple of similar projects which serve a similar purpose but achieve it slightly differently.

• opam2nix is the original in this space. However, it requires committing generated files into the repository, doesn’t integrate well with Flakes, and is not as flexible.
• opam-nix-integration is quite similar to opam-nix. However, it’s not as flexible since it doesn’t allow to import opam switches or easily call multiple packages from the same workspace.

Conclusion

opam-nix is a flexible yet user-friendly library to turn opam packages into Nix derivations. It is already mature enough to be used by dune, ocaml-lsp, and others, and yet there are still features and interface improvements waiting to happen. We encourage you to try it on your project and share your experience and feedback in the issue tracker.

Why would I want to run a NixOS virtual machine on macOS?

My colleague at Tweag, Dominic Steinitz, asked me this question after I shared my first minor achievement in this area, and it struck me that I have never described why exactly I run virtual machines (VMs) on my laptop and why I want to make it easier for myself (and others).

Like many members of the Nix community – more than 25% according to the Nix community survey – I rely on macOS to provide me a stable shiny desktop environment. However NixOS is based on Linux and so many pieces that are used for developing and testing it require running it in VMs. I’ll describe some major use cases for it below.

Local remote builder

macOS is a great desktop environment, but as soon as you move beyond your machine, be it to a remote server, VM or container, you most likely end up in Linux, which is a different system. Practically speaking, this means that a derivation for any of them requires a Linux machine with Nix to be built.

Therefore, you have to set up a remote builder, either on some other machine (in the cloud, some datacenter, or under your desk), or in a VM on your machine. If you have a powerful enough machine, you might want to opt for the latter, reaching for either running a complete Linux VM in VirtualBox, UTM, Parallels or vmWare, or using nix-docker to run it in Docker’s VM or linuxkit-nix to spawn a minimal Linux VM with only Nix on board.

Most of these options require you to install some app that will manage VMs for you and most of those are proprietary. As for linuxkit-nix, Gabriella Gonzales wrote in her blog post a great overview of how it is lacking in different areas.

Test NixOS configurations locally

Now that you can build and run packages for your target Linux system, you might want to start deploying NixOS to remote machines. When you are running Linux locally, you can easily run nixos-rebuild build-vm to generate a script that will start a local QEMU virtual machine. This allows you to verify if the configuration works without affecting any actual system. It is very lightweight because it mounts your /nix/store directory directly into the VM and boots the VM straight from it, with no additional image overhead. You can even rely on it to wrap some services into a VM and configure it to run as a daemon on your host by using the system.build.vm attribute of any NixOS configuration. It uses a NixOS configuration variant from virtualisation.vmVariant that imports the virtualisation/qemu-vm.nix module to build the VM.

However, on macOS none of that will work. nixos-rebuild by default makes the assumption that the local machine is Linux and links to Linux Bash and QEMU binaries, so you can’t run it locally. Hence, an alternative approach is needed.

Run NixOS tests locally

NixOS provides a powerful framework for testing different services and modules in VMs. You just provide it with a piece of NixOS configuration for all machines that you need and it provides a neat API to run them, interact with them and check their state. This framework is based on the same mechanism as other NixOS VMs, so it doesn’t work on macOS out of the box either. Here, we also need an alternative.

What has been done?

Since the conventional means of working on NixOS configurations relies on a Linux host, multiple changes to the status quo are needed to make it comfortable to work on them on macOS, presented in the following sections.

Build VMs from NixOS configurations

The first issue that needed to be fixed is the lack of a simple interface for building and running NixOS VMs on macOS. Starting with a Linux host before migrating our workflow to macOS, you could have a NixOS configuration defined in a flake like this:

{
  outputs = {
    self,
    nixpkgs,
  }: {
    nixosModules.base = {pkgs, ...}: {
      system.stateVersion = "22.05";

      # Configure networking
      networking.useDHCP = false;
      networking.interfaces.eth0.useDHCP = true;

      # Create user "test"
      services.getty.autologinUser = "test";
      users.users.test.isNormalUser = true;

      # Enable passwordless ‘sudo’ for the "test" user
      users.users.test.extraGroups = ["wheel"];
      security.sudo.wheelNeedsPassword = false;
    };
    nixosConfigurations.linuxBase = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules = [
        self.nixosModules.base
      ];
    };
  };
}

Note that this configuration lacks some key settings, like filesystems or bootloader configuration required for it to be actually deployed, but there is just enough to build a VM out of it.

Still on Linux, you can use nixos-rebuild to create a VM using this configuration and run the result:

$ nixos-rebuild --flake .#linuxBase build-vm
building the system configuration...
warning: creating lock file '.../flake.lock'

Done.  The virtual machine can be started by running /nix/store/3ad15806lclvmfzxkppabncp5sq9i93n-nixos-vm/bin/run-nixos-vm
$ ./result/bin/run-nixos-vm
Formatting '.../nixos.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=1073741824 lazy_refcounts=off refcount_bits=16

It will start a VM, with an external window emulating its graphical terminal screen, where the test user will be automatically logged in and at their shell prompt. It’s easier to follow this article along without switching between the terminal and external windows, so let’s change that in a separate module specific to the VM in our flake. We’ll instruct vmVariant of our NixOS configuration to not use graphics for virtualisation. That would force it to use a text based serial terminal output linked directly to the terminal from which we’re running the script.

{
  outputs = {...}: {
    # ...
    nixosModules.vm = {...}: {
      # Make VM output to the terminal instead of a separate window
      virtualisation.vmVariant.virtualisation.graphics = false;
    };
    nixosConfigurations.linuxVM = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules = [
        self.nixosModules.base
        self.nixosModules.vm
      ];
    };
  };
}

Now let’s build it with nixos-rebuild to run it in our terminal this time:

$ nixos-rebuild --flake .#linuxVM build-vm
building the system configuration...
warning: Git tree '.../blog' is dirty

Done.  The virtual machine can be started by running /nix/store/f9a3ac5ydcm6anv9zpf2sqykzivwccaw-nixos-vm/bin/run-nixos-vm
$ ./result/bin/run-nixos-vm
SeaBIOS (version rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org)
...
[test@nixos:~]$ uname -a
Linux nixos 5.15.74 #1-NixOS SMP Sat Oct 15 05:59:05 UTC 2022 x86_64 GNU/Linux

[test@nixos:~]$ sudo poweroff
...
[   75.762105] reboot: Power down

Note that you can use sudo poweroff to turn the machine off, or the Ctrl-A X QEMU shortcut. The latter might leave your terminal in a weird state, so you might need to run reset to fix it.

Also note that I’ve created a Git repo and added flake.nix and flake.lock files to it with git init && git add flake.*. Outside a Git repository, Nix considers all files in the directory (including any temporary files, like nixos.qcow2 or result) relevant to the flake and pulls them into the build process. When Nix detects that its target is a Git repository, it only considers files that Git knows about and ignores all untracked files.

So far we’ve been using nixos-rebuild to build our VMs, but it doesn’t provide flexibility to build on other systems. Since we’ve imported the qemu-vm.nix module, we can use its output to run the VM instead:

$ nix run .#nixosConfigurations.linuxVM.config.system.build.vm
warning: Git tree '.../blog' is dirty
SeaBIOS (version rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org)
...

We could also add it as a package to our flake so that we don’t have to spell it out every time:

{
  outputs = {...}: {
    # ...
    packages.x86_64-linux.linuxVM = self.nixosConfigurations.linuxVM.config.system.build.vm;
  };
}

Now we can run it with a short command:

$ nix run .#linuxVM
warning: Git tree '.../blog' is dirty
SeaBIOS (version rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org)
...

So far we haven’t been doing anything that hasn’t been possible on any old version of Nixpkgs. Let’s move to our macOS machine and try running this VM:

$ nix run .#linuxVM
warning: Git tree '.../blog' is dirty
error: flake '.../blog' does not provide attribute 'apps.aarch64-darwin.linuxVM', 'packages.aarch64-darwin.linuxVM', 'legacyPackages.aarch64-darwin.linuxVM' or 'linuxVM'
$ nix run .#nixosConfigurations.linuxVM.config.system.build.vm
warning: Git tree '.../blog' is dirty
error: a 'x86_64-linux' with features {} is required to build '/nix/store/297gh5gn4ihnd0av0qbfx14vg0azly8x-append-initrd-secrets.drv', but I am a 'x86_64-darwin' with features {benchmark, big-parallel, hvf, nixos-test}

We don’t have an output named linuxVM that would work on aarch64-darwin, and even if we were to make it, it would require a Linux builder to be built. You can spin up a NixOS VM somewhere in a cloud or on your local machine and configure your Nix daemon to use it. You can run it locally using one of the methods described above or read on to the next section for a new one.

Assuming you have a Linux builder configured, let’s go one step further:

$ nix run .#nixosConfigurations.linuxVM.config.system.build.vm --options builders ssh-ng://some-linux-builder
warning: Git tree '.../blog' is dirty
/nix/store/f9a3ac5ydcm6anv9zpf2sqykzivwccaw-nixos-vm/bin/run-nixos-vm: line 7: /nix/store/4vjigg3pr8bns6id4af51mza5p73l9lx-coreutils-9.1/bin/readlink: cannot execute binary file

The error output suggests that you cannot run this on macOS, likely because this output is only expected to be run on Linux.

Looking for a solution for this I found an old issue where Silvan Mosberger started a discussion about running NixOS VMs on macOS and developed a prototype solution. It allows us to specify a different package set for the host where the VM is supposed to be running. Several patches to support using sharing directories from the host and running on Apple Silicon hardware have landed in QEMU master since, and using the 7.0.0 release plus a couple patches it finally worked. I’ve adapted the prototype and backported the required QEMU fixes to 7.0.0 in Nixpkgs.

With all these changes, we can specify virtualisation.host.pkgs to run a VM directly on macOS:

{
  outputs = {...}: {
    # ...
    nixosConfigurations.darwinVM = nixpkgs.lib.nixosSystem {
      system = "aarch64-linux";
      modules = [
        self.nixosModules.base
        self.nixosModules.vm
        {
          virtualisation.vmVariant.virtualisation.host.pkgs = nixpkgs.legacyPackages.aarch64-darwin;
        }
      ];
    };
    packages.aarch64-darwin.darwinVM = self.nixosConfigurations.darwinVM.config.system.build.vm;
  };
}

Note that I’ve changed it to aarch64-linux because I’m running on Apple Silicon and want to take advantage of its hardware assisted virtualisation. I’ve also set virtualisation.host.pkgs for vmVariant of this NixOS configuration that is used to build the VM. Now let’s run it:

$ nix run .#darwinVM
warning: Git tree '.../blog' is dirty
Formatting '.../blog/nixos.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=1073741824 lazy_refcounts=off refcount_bits=16
[    0.152640] armv8-pmu pmu: hw perfevents: failed to probe PMU!
...
[test@nixos:~]$ uname -a
Linux nixos 5.15.74 #1-NixOS SMP Sat Oct 15 05:59:05 UTC 2022 aarch64 GNU/Linux

[test@nixos:~]$ sudo poweroff
...
[   11.565895] reboot: Power down

Bootstrapping a local builder for Linux

As I mentioned before, to build any Linux configuration on macOS you need a remote Linux builder. There are some existing options for that: nix-docker that relies on Docker running and essentially providing a Linux VM on its own, or linuxkit-nix that creates a custom Runit-based system with very little room for customisation.

With the pull request providing virtualisation.host.pkgs option merged, Gabriella Gonzales created a pure NixOS-based builder, which has been merged into Nixpkgs. Now you can start using the local builder straight from the upstream Nix cache, then modify its configuration and rebuild it locally. See Gabriella’s blog post for more information.

Running NixOS tests on macOS

There is an overarching issue on running NixOS tests on different virtualisation technologies on different platforms. Robert Hensing summarises the latest changes in this area pretty well in his comment. They include his work on making NixOS tests modular and a draft implementation of providing an option to run them on macOS.

What else can be done?

Making nixos-rebuild aware of Darwin

On Linux nixos-rebuild allows you to build a NixOS system configuration, run it in a VM or apply it to a remote NixOS system. On Darwin you can use it to build a configuration, although it’s hard to get man pages that are distributed with NixOS itself, rather than in the nixos-rebuild package. You can also build a VM with nixos-rebuild build-vm, but it will only run on Linux. We could detect what platform nixos-rebuild is running on and provide a VM compatible with said platform or allow the user to specify it.

When managing remote NixOS systems from macOS, you have to be aware that all derivations have to be built on a Linux platform, so you might want to put your remote system in --build-host to avoid having to fetch all dependencies locally, then copying dependencies and derivations to the builder, then copying the NixOS configuration with all its dependencies from the builder to the local machine, and finally copying them to your remote system. You also have to use --fast because by default nixos-rebuild builds Nix and itself from the NixOS configuration that you provide to it, and those are Linux binaries. With all these parameters on macOS you have to specify a command line like this:

$ nixos-rebuild --build-host user@remote-host --target-host user@remote-host --use-remote-sudo --fast ...

We could detect that nixos-rebuild is not running on Linux and default to building on the target host and either not use newly built binaries at all or build them for the platform it is running on.

Reducing number of system-dependent derivations

If you try running nixos-rebuild dry-build on a new NixOS configuration, you will see a lot of paths that will be fetched from the cache, but also at least 200 derivations that will be built, all requiring Linux to do so. It would seem logical that a Linux system requires Linux packages, but all packages are already built by Hydra and available from the cache. All these derivations produce different configuration files like systemd services, symlink trees like /etc and some shell scripts like nixos-rebuild itself, that are built by substituting a bunch of strings in a text file. Aside from being penalised for using mkDerivation for such simple steps, all those actions don’t really depend on the platform they are running on. Nix requires us to guarantee that the result of the derivation will not change by specifying a concrete platform and a concrete builder that will run on that platform, which is Bash in all these cases. We could lean onto some virtual machine to handle this guarantee and run the same builder binaries on all supported platforms instead. The current idea is to provide a wasm32-wasi platform that will run builders compiled to WASM, and allow building most of those derivations locally.

Conclusion

We want to make macOS a first class citizen in the Nix ecosystem, and this post shows some of the weaknesses that macOS users face when working with Nix and NixOS. We as a community should strive to overcome these difficulties, and it’s great to see a lot of progress in this area. There’s still a lot to be done though and I hope to be able to help moving this forward.

Why the name change? We want to reach a bigger audience. If you are a regular Nix user, jupyterWith is a good name. It is analogous to the python.withPackages function in nixpkgs. But outside of the Nix ecosystem, the with naming scheme is non-existent. To better target people outside the Nix ecosystem, we are changing the name that emphasizes Jupyter environments (hence jupyenv).

New API

Module System

jupyenv now uses NixOS modules to configure the JupyterLab environment and kernels. The previous API update reduced complexity, but using modules reduces it even further without sacrificing capability. A minimal working Python kernel used to be configured like so.

# Old interface
{
  availableKernels,
  name,
  extraArgs,
}:
availableKernels.python {
  name = "${name}-example";
  inherit (extraArgs) system;
}

While this is not terribly complicated, it exposes the user to the internal machinery of jupyenv. What is availableKernels? What is extraArgs and what else can I pass in? Is there something special about the name attribute? (Spoiler: Yes! They have to be unique and without spaces.)

Now with modules, we can hide all the inner workings. The following is equivalent to the previous example, but uses the new modules API.

# New interface
{...}: {
  kernel.python.example.enable = true;
}

Using modules not only simplifies configuring kernels, but it also simplifies how they can be organized. You are no longer required to have separate kernel directories. You can if you prefer, but using modules provides the flexibility to configure all the kernels in one location.

The following is a working example configuration with two different Python kernels, a Bash kernel, and an OCaml kernel.

{...}: {
  kernel.python.minimal.enable = true;

  kernel.python.science.enable = true;
  kernel.python.science.extraPackages = ps: [
    ps.numpy
    ps.scipy
  ];

  kernel.bash.minimal.enable = true;

  kernel.ocaml.science.enable = true;
  kernel.ocaml.science.ocamlPackages = {
    hex = "*";
    owl = "*";
  };
}

In the previous examples, we used names like example, minimal, and science to configure a specific kernel. Those names are purely descriptive and used so you can configure multiple kernels of the same type (e.g. python.minimal and python.science), and have them both available in your JupyterLab environment. For more information, see the Kernels section on the How To page.

Documentation

One of the perks of using NixOS modules is their ability to create documentation. Though the previous update was an improvement, the user would have to read the source code to find what arguments were available and how to use them. Ideally, documentation would be available on the site, but it cannot be done manually, since it would quickly go out of date. Now, with modules, the documentation is part of the option definitions and automatically uploaded to the jupyenv site as it changes.

We have taken great care to ensure the new modules documentation is tidy and easy to navigate. With a bit of pre-processing and styling, child options are nested under their parents making the structure clearly visible. Additionally, with a small amount of JavaScript, each option can be expanded. You can see a demo of the new Options documentation in the video below.

If you are not a fan of JavaScript or use a Text-Mode Web Browser, the page will still render nicely. It is also navigable without the use of a mouse and should have the relevant context for screen readers.

Blog

With the last API update, we found it was difficult to inform our users that things were changing. Sure, there is the Tweag blog, and it does well for major announcements, but there are times when we want to inform our users without having to write a major blog post. To keep users in the loop, we have added a Blog tab to the site. Here we will post about new releases, interface changes, and bug fixes.

Releases

The aforementioned new releases are going to be available as Git annotated tags in our GitHub repository It is not so useful to mention a new release without some reference to a point in time in the repository. So to help users not only see what changed, but when also when it changed, we will begin versioning jupyenv and announcing it on the blog.

Community

We want to create a community of jupyenv users, but so far there was no space for them to communicate directly. Where can they ask questions? Where can they collaborate with other users? To answer this, we have created a Matrix Space which you can find in the Community tab on the site or here.

Conclusion

There are a lot of new updates. A new name, a new module based API, updated and improved documentation, a blog, tagged releases, and a matrix space. If you are a current jupyenv user, we hope the changes improve your user experience. If you have never tried it but are interested in reproducible Jupyter environments or reproducible data science, please give it a try. Join us on the journey to make Jupyter reproducible.

At Tweag, we are working on Nickel, a configuration language featuring a gradual type system. Defining Nix derivations, NixOS modules or flakes using Nickel would catch more mistakes, catch them earlier and identify their precise location.

However, the Nix language is used to define more than 80,000 packages,¹ to which one could add all flakes, NixOS modules and Home Manager configurations. For all this historical effort to not go to waste, Nickel code needs to be able to leverage Nixpkgs.

We can either make Nix able to call Nickel code, or the other way around. We chose the latter: make the Nickel interpreter able to evaluate Nix code. Indeed, making Nix evaluate Nickel code would negate its benefits, namely, its error reporting, the integrated documentation and the nice way records are merged with priorities.

This comes with several technical challenges. One being the handling of the widely used, yet controversial, with construct.

In this blog post, we’ll explain the challenges of calling Nix from Nickel, specifically for expressions which use the with construct.

The challenge of calling Nix from Nickel

Evaluating Nix presents two contradictory challenges. On the one hand, calling Nix from Nickel should be done without breaking the safety provided by the Nickel type system. Nix is dynamically typed, but one purpose of the gradual type system of Nickel is precisely to make statically typed code and dynamically typed code interact gracefully. On the other hand, we want to update the core of Nickel only when strictly necessary, because Nickel is not limited to targeting Nix and NixOS. Fortunately, Nickel is almost a superset of Nix, so a balance between these objectives can be found.

To evaluate Nix expressions in Nickel, we can use either an FFI or code translation (a.k.a. transpilation). We decided to go for the latter: translating Nix code into Nickel code, which can then be evaluated. Indeed, since the Nix and Nickel languages are built on similar foundations (loosely, a lazy JSON with functions), translating from Nix’s AST to Nickel’s is almost seamless. Such translation also allows us to implement special features specific to this use case. For instance, Nixpkgs annotates types as comments for most of its functions, so we could infer types from these. Moreover, we estimated that implementing an FFI system would be too complex for what we wanted to achieve here. Thus, we made a Nix to Nickel transpiler.

However, some constructions weren’t so straightforward to translate:

• the inherit keyword (which will probably be one of the Nix constructs added to Nickel’s core.²), and

• the with keyword.

In addition, none of the standard library built-ins can be evaluated at the moment, as most of them don’t have an equivalent in Nickel.

Nix with is useful, but confusing

In Nix, the with keyword is used to bring all the fields of a record in scope.

Let’s consider a simple Nix expression.

{ pkgs, ... }:

{
  packages = [
    pkgs.firefox
    pkgs.thunderbird
    pkgs.libreoffice
  ];
}

As you can see, all the elements of the list are accessed via pkgs. This looks quite repetitive, and is arguably harder to read. We can use with to make it clearer.

{ pkgs, ... }:

{
  packages = with pkgs; [
    firefox
    thunderbird
    libreoffice
  ];
}

Such Nix expressions can be found anywhere, from Nixpkgs to your own NixOS configuration. Thanks to the with construct, you don’t have to prefix each item with pkgs.. That is quite convenient.

So what could be the issue here?

At first, one might think that the with construct is syntactic sugar that statically prepends record field access (here pkgs.) to some identifiers. Unfortunately, the way it actually works is more complex than that.

To demonstrate some possible issues, let’s look at a more complex Nix expression.

let
 env = {
    linux = {name = "linux-env";};
    system = {name = "system-env";};
  };
  lib = {
    linux = {name = "linux-lib";};
    systemd = {name = "systemd-lib";};
  };
  linux = "x86_64_linux_gnu";
in

with env; {
  system = system.name;
  deps = with lib; [
    linux
    system
  ];
}

What would this evaluate to? Try to think about it, then check the answer below.

let
 env = {
    linux = {name = "linux-env";};
    system = {name = "system-env";};
  };
  lib = {
    linux = {name = "linux-lib";};
    systemd = {name = "systemd-lib";};
  };
  linux = "x86_64_linux_gnu";
in

with env; {
  system = system.name; # "system-env"
  deps = with lib; [
    linux        # We get the one defined in `let`, not lib.linux,
                 # which is "x86_64_linux_gnu"
    system       # It's a typo! We wanted lib.systemd, but we get
                 # env.system instead of an error
  ];
}

Did you get it right? Probably not!

When with blocks are nested, the behaviour can be confusing.³

Finally, as you may have noticed, we can’t know before evaluation which field contains the record passed to with. This is due to Nix’s dynamic typing. Because of this behaviour, we encounter something that looks like standard (static) variable access, but in reality, is dynamic record access. In code where one would expect “unbound variable” errors, Nix will instead throw runtime errors that are generally tricky to debug.⁴ Moreover, these errors cannot be caught by an LSP, so they cannot be shown in a code editor. For the same reason, we can’t provide auto-completion hints within a with block, at least not without hacks that perform evaluations.

Transpiling with to Nickel

Because of the problems explained above, the Nickel team decided to not implement any with-like operator in the interpreter. At least not with the exact same behaviour. Regardless, for compatibility with Nix, we should find a way to evaluate it anyway.

What we propose in this article, which will be implemented in the near future, is a mix of Nickel code generation and compatibility function calls. To detail this, step-by-step, let’s revisit our simple example from earlier:

{ pkgs, ... }:

{
  packages = with pkgs; [
    firefox
    thunderbird
    libreoffice
  ];
}

This will be translated to something like:

fun { pkgs, .. } =>

{
  packages = [
    compat.with [pkgs] "firefox",
    compat.with [pkgs] "thunderbird",
    compat.with [pkgs] "libreoffice",
  ]
}

You will notice two things on the compat.with call:

1. The array [pkgs] is the first parameter. We will detail why later, but it has to do with nested with.

2. The field being looked for is passed as a string and not a static identifier. That’s because, at parse time, we can’t know which record contains which fields. Indeed, in Nickel, every variable access is checked statically; the usage of an identifier instead of a string here would throw an “unbound variable” error.

An implementation of the compat.with function might look like:

with
  : Array {_: Dyn} -> Str -> Dyn
  = fun envs field => (
    array.fold (fun current acc =>
      if !acc.found && record.has_field field current
      then { value = current."%{field}", found = true}
      else acc
  ) {found = false, value = null} envs).value

The function folds on the first parameter, envs, which is an array of with records. This array has to be ordered from the outermost to the innermost with block; the outermost being the head of the array. This order is justified because the more internal a with, the higher its priority. Please note that fold is a right folding. If multiple records contain field, the rightmost one is returned by the fold, that is, the innermost one. In the case where none of them have this field, the initial value of fold ({found = false}) is returned. Finally, we try to access the field value. This last access operation will differ in the final implementation: it will probably use a contract application to assert on found.

This operation will fail only if {} is returned. In other words, only if none of the records in envs contain the required field.

That’s all for the stuff that happens at runtime.

For what is done at parse time, we have:

• let-defined variables,
• functions parameters, and
• fields inside a recursive record.

Let’s revisit our more complicated example:

{ env, lib, ... }:

let
linux = "x86_64_linux_gnu";
in

with env; {
  system = system.name;
  compatible = system == linux;
  deps = with lib; [
    linux
    system
  ];
}

It will be translated as follows, during the parse phase:⁵

let env = {
  linux = {name = "linux-env"},
  system = {name = "system-env"},
}
in

let lib = {
  linux = {name = "linux-lib"},
  systemd = {name = "systemd-lib"},
}
in

let linux = "x86_64_linux_gnu" in

let _with_ = compat.with [env] in
{
  # NOTE Even though records, in Nickel, are recursive by default,
  # the record in this AST is not.

  # No var named "system" found, so substitute by a call to __with__
  system = (_with_ "system").name,

  deps = let _with_ = compat.with [env, lib] in [
    linux,              # Nickel statically found a defined var named "linux" so didn't substitute
    _with_ "system",  # "system" isn't found in lib so it takes the env field
  ]
}

To perform this translation in our Rust implementation, we pass around two extra things: a list of in-scope variables, and a stack of with records. The first is updated by inserting the identifiers when entering any of:

• a let binding body,
• a function definition, or
• a recursive record.

When leaving these blocks, the list has to be reset to the state it was in before entering. The with stack is updated during a with block translation: we push the identifier at the beginning and pop it at the end.

Finally, for each variable translation, we perform these steps:

1. We check the existence of the variable identifier in the list of variables. If we find one, or if the with stack is empty, the variable is translated to a Nickel variable.

2. If the variable has not been defined before, and if the with stack is not empty, the variable is translated to a compat.with call with the with stack as its first parameter, and the identifier stringified as its second.

By checking the with stack as described above, we keep the undefined variable errors at type checking time when we are not in a with block. This is more or less how Nix evaluates with inside its interpreter. We only rewrite the static part into Rust and the runtime one with Nickel.

The future of with in Nickel

As we discussed, Nickel cannot avoid the use of dynamic fields access to emulate Nix with. So then, what could a Nickel-friendly with look like? Moreover, given Nickel’s type system and the fact that it already has a way to destructure records, do we really want a Nickelized with?

Remember the two main issues regarding Nix with:

• shadowing or, more accurately, not shadowing of already-defined variables; and
• the fact that the variables made available by with are unknown statically.

To handle the first issue we could make with overload already-defined variables. Alternatively, we could throw an error if we call with on a record containing a field that shares a name with a variable in scope. Without knowing the fields of the record, the first option remains confusing, while the second is not possible.

In Nickel, the only way to know for sure that a record contains a field is to rely on the type checker. So a correct with can exist in Nickel only if it requires a statically typed record, where the fields we want to use must also be typed (e.g.: myrec: {field: Dyn, otherfield: Num}). Obviously, a type like {_: SomeType} does not respect this rule because fields are not statically defined.

That said, the use of a closed record contract is possible. In this case, we can see the contract as a dynamic cast. The power of this approach is to permit static typing and auto-completion in the with block. This goes a long way towards fulfilling the necessity of statically knowing every record’s fields, and focuses the possibility of evaluation errors to the position of the cast. These errors are clear enough (e.g. missing or extra fields). In this manner, a Nickel with would behave like an import or an open in languages with statically typed modules (Rust, OCaml, Haskell, etc.).

Conclusion

This is only the beginning of the Nix/Nickel story. For now, Nickel is only able to evaluate Nix expressions which do not use any standard built-ins or inherit, and the error reporting is not always as detailed as Nix’s.

The question of Nickel with is far from settled. It is clearly useful, when we need to bring a lot of fields from a record into scope, but the requirement for types may only end up moving the extra code to the type annotation, from the destructuring pattern. Moreover, at least two proposals for Nix (RFC 110 and RFC 120) open the possibility of a future deprecation of with. Their proposed syntax is a better way to write the most common use case of with, so that may also be a direction that we consider for Nickel.

In summary, we are not totally rejecting the idea of a Nickel with, but ultimately, it will be really different from its Nix cousin.

Bazel is a fantastic build system which by itself provides many benefits. However dependency management could be considered an area of weakness for it. This is understandable given its monorepo origins at Google but when not working in a monorepo, what is the best way to provide the dependencies a project needs?

One approach is to simply assume the dependencies exist in the environment (having been put there by apt install or the like), but this is just ignoring the problem. Bazel has the concept of external repositories and is able to fetch and build source code from a variety of places¹. Under some circumstances this might be a reasonable solution, but we probably don’t want to spend significant amounts of time building a large dependency tree for a project that itself only takes a few minutes to build.

Instead, Nix can be used to supply the dependencies to Bazel prebuilt. Nix and Bazel are very similar technologies (both build code from source and both place strong emphasis on properties like determinism and hermeticity), but without delving into the details it is perhaps best to think of Nix as a package manager and Bazel as a build system. The advantage of Nix is that while it ostensibly builds everything from source, in practice it usually just downloads artifacts from the Nix binary cache.

The project I migrated is an unfinished game with around fifteen thousand lines of C++ code. It was originally built using some very convoluted makefiles and is structured as follows:

This game has a few attributes that make it an interesting case study:

• It has a non-trivial structure with modules² shared between client and server executables.

• It depends on several third party libraries. Previously these were manually built from source. Now with the help of rules_nixpkgs they are provided by Nix. Typically this means they will be fetched prebuilt from the Nix cache, saving the time it would have taken to compile them.

• In addition to code that needs to be built, the game has assets like 3D meshes and textures that need to be transformed into a format that can be loaded at runtime.

• One particular transformation is handled by the custom zonebuild tool. If any of the source code that goes into this tool changes and it gets rebuilt, the data transformation it performs also needs to be rerun.

In a more traditional build system it is difficult to express a build graph for the above in a way that gives perfectly correct and incremental builds. Things tend to be more ad-hoc and changes to tools or across language boundaries are usually best dealt with by cleaning and rebuilding from scratch.

Building the source code

Every Bazel project requires a WORKSPACE.bazel file at its root. As the migration advances this file will grow in complexity, but to begin with it simply specifies the workspace name:

workspace(name = "space-game")

A Bazel workspace is organised into packages by BUILD.bazel files. There are many ways you might do this (e.g. a separate package for each module and executable) but for a project of this scale it is reasonable to build all the source code from a single package named //common/src.

As it has no dependencies, building the core module is a good place to start. It just requires a cc_library rule in common/src/BUILD.bazel:

cc_library(
    name = "core",
    hdrs = glob(["core/*.hpp"]),
    srcs = glob(["core/*.hpp", "core/*.cpp"]),
    copts = ["-std=c++17"],
    includes = ["."],
    visibility = ["//visibility:public"],
)

At this point it is possible to build the core module by running the following command:

bazel build //common/src:core

Bazel does not pollute the source tree with intermediate build files and output files. These are instead created within the outputRoot directory and some convenient symlinks are added to the workspace root. If you follow the bazel-bin symlink you’ll find that cc_library has produced both static and shared libraries:

bazel-bin/common/src/libcore.a
bazel-bin/common/src/libcore.so

This is pretty cool. In the old days, linking even a moderately complex project was quite a tricky affair. You had to determine which parts it made sense to link statically and which parts dynamically. Code going into a shared library needed to be compiled with -fPIC because shared libraries might be loaded anywhere in process address space. Getting something wrong usually led to unhelpful linker errors about missing or duplicate symbols. With Bazel all those messy details have been abstracted away.

Next let’s examine how the net module is built. If you check the graph above you’ll see that net depends on core and two third party libraries, boost and enet. As mentioned in the introduction, these libraries will be fetched from Nix. First we need a nixpkgs.nix file:

let nixpkgs = fetchTarball {
  url = "https://github.com/NixOS/nixpkgs/archive/ce6aa13369b667ac2542593170993504932eb836.tar.gz";
  sha256 = "0d643wp3l77hv2pmg2fi7vyxn4rwy0iyr8djcw1h5x72315ck9ik";
};
in import nixpkgs