Stack and Cabal are great tools for incremental development. But before even starting to think about what your first commit will be about on a freshly checked out project, what you have to do is get a working development environment, by which we mean: see if it builds! Stack improves your chances that the build will Just Work(tm) by leaps and bounds thanks to its focus on reproducibility, but if your project has (non-Haskell) system libraries as dependencies then the build may well still fail. On a team project with lots of developers, you don’t want to be wasting everyone’s time by having everyone setup their system just right before launching the build. And if it’s an open source project, you’d be gutted if anyone got discouraged from contributing a little patch or two due to the build process being an error-prone multi-step ordeal.

What goes in the development environment, apart from Hackage dependencies, is the set of tools and system libraries the project depends on. Current build tools such as cabal-install and stack check that these are installed somewhere globally on your system, but they can’t automatically install anything themselves. And for good reason: they are not package managers, merely build tools.

What we are proposing here is to let Stack manage building the Haskell dependencies as it always has, but teach it how to build ad hoc environments where all the system dependencies are guaranteed to be present. Turns out the “teaching” part was easy: we got Stack to reuse Nix, an existing off-the-shelf toolchain for creating ad hoc build environments. So easy in fact that this feature has already shipped as part of Stack’s latest release.

What this means in practice is: you can build an adequately configured Stack project replete with all manner of system dependencies and be pretty sure it’ll all just work, no matter whether you already have conflicting versions installed on your system. Yet we haven’t turned Stack into a package manager. Read on to find out why.

Why Nix ?

Nix is a multifaceted tool. A build system, a package manager, a lazy dynamically-typed functional language, you name it. In our case, what interests us is its capacity to express the dependencies and the build of a project declaratively, to automate the download of dependencies as any good package manager would and to provide a local environment (the nix-shell) in which commands (build operations in this situation) should be run, as any Linux container would.

However, Nix isn’t a package manager in the usual sense. For one, Nix isn’t distro specific: it supports all Linux distros, OS X and in principle Windows too. Further, it’s purely functional, meaning that it never installs anything globally at a named location, since this would amount to having the side-effect of mutating the state of your global /usr directory. So really you can think of Nix as a lightweight containerization technology: given a declarative specification of what packages and configuration you want, it copies package content somewhere (doesn’t matter where) and then sets up your local shell so that these packages (and only these packages) are available inside it.

Compared to other containerization technologies such as Docker, Nix does less. It’s a more lightweight solution that does not virtualize every namespace out there such as the process namespace, the network namespace, the user namespace, the mount namespace, etc. It just does the deed with a clever use of symlinks under the hood. So you’re getting weaker isolation guarantees with Nix than with Docker (which Stack already supports). On the flipside though:

• for simply building an open source project, you’re often not interested in iron-clad isolation - you just want the build to work, one way or another;
• Nix’s way of achieving isolation between projects and from the system uses only regular POSIX filesystem calls, so it’s a more portable solution that works on several platforms;
• Nix is better at sharing the storage space on disk: the unit of sharing is derivations (i.e. “packages” to some approximation), rather than coarse-grained and seldom shareable layers on top of full distro images. So you have less to download and you get to spare more disk space.

A cool feature of Nix is that if you need a few packages available, just say so and it will download them automatically if needed, yet reuse whatever was already downloaded from previous invocations if possible. For instance,

$ nix-shell -p ghc cabal-install haskellPackages.hasktags

will drop you in a shell with a bare bones environment for hacking Haskell using GHC and cabal-install. It’s instantaneous the second time you do that.

Why not just use Nix directly, then?

Nix is not a tool for incremental development, nor does it aims to become that. We thought, “leave it to Stack to do what it does best, i.e. (re)building Haskell projects”. Incremental recompilation across multiple packages at once, fine-grained parallelization, etc are all best done by a dedicated tool, that by now all Haskeller developers know best and works extremely well.

Further, we wanted to afford the users the benefits of lightweight reproducible builds without having to burden anyone with learning how to use a whole new toolchain. We designed Stack’s Nix support so that you don’t have to learn any Nix command at all to get started, or indeed any new language. If you have a complex project with complex needs, by all means describe your system dependencies using Nix’s very powerful domain-specific language for doing so. But it’s certainly not a requirement for beginners.

An example

Let’s get down to business. As an example, we will show how to use Stack’s new Nix support to hack on a project that uses GLPK, a C library popular in numerical computing, via the Haskell bindings provided by glpk-hs. You’ll need to follow the install instructions for Nix first if you don’t have it already, or this one line for the non-paranoid:

$ curl https://nixos.org/nix/install | sh

Use the command stack new foo, as normal, to prepare a brand new project. Go into the directory foo/ and edit the file stack.yaml so it looks like this:

packages:
  - "."
extra-deps:
  - glpk-hs-0.3.5
resolver: lts-3.7
nix:
  enable: true
  packages: [glpk]

Add a dependency to glpk-hs in your project’s only .cabal file. Building this project, including the system dependency, is a case of

$ stack build

No --extra-include-dirs or --extra-lib-dirs to get right. You can also set

nix:
  enable: false

if you prefer to make using Nix under the hood explicit to provision the GLPK C library, as in

$ stack --nix build

Note that you don’t need to have GHC installed on your system. Stack will as usual manage all the details for you. Under the hood, when Nix support is enabled Stack will actually download GHC using the Nix toolchain, choosing the version that matches the resolver you set. So you’ll need to use a resolver Nix knows about, i.e. that has been imported in the nixpkgs package collection. To find out, look here and find out whether a configuration-lts-X.Y.nix file exists for say resolver LTS-X.Y.

For more information regarding Stack command-line options that are specific to Nix, run

$ stack --nix-help

There’s also a dedicated section in the Stack Guide.

How it’s done

The design of this new feature for Stack has been heavily inspired by the design of the Docker support, as presented in this post by Emanuel Borsboom. The configuration of the Nix-shell backend in Stack also mirrors that of Docker. It works by having Stack relaunch itself inside a nix-shell. The implementation has fewer details to deal with, because unlike the Docker support there is no bind mounting of the project content inside an ephemeral container to deal with etc. In theory, you could even activate both Docker and Nix support at the same time (which would result in stack provisioning system dependencies from within a Docker container).

Concluding words

So have we just made Stack a full-fledged package manager for all things Haskell and beyond? We’d argue, not quite yet, fortunately! Because Stack did not and still does not fiddle with any system-global or user-global resource. It’s just that it will automagically provision system dependencies locally for your current project. It already knows how to do that via Docker. All we did was teach it a different way of doing the same thing that trades a few isolation guarantees for more portability to work on OS X and elsewhere.

In fact one of the original motivations for embarking on this project was building HaskellR reliably. This is a sizable multi-package project with several system dependencies (R, Jupyter, ZeroMQ, etc) so users were bound to encounter difficulties when building on their system. We anticipated those problems by leveraging Stack’s support to build inside a dedicated Docker image. But OS X users were complaining that they wanted fast, reliable builds too!

Enabling build tools to locally provision system dependencies raises an interesting question for the community. Our first step in this direction was to add this feature to Stack, to locally provision the dependencies for projects. We added new metadata to the project metadata file, i.e. stack.yaml. But as discussed with Neil Mitchell here, arguably it ought to be possible to make system dependencies package metadata. That is, included in the .cabal file. That way, packages can hide their system dependencies, arguing that it’s an implementation detail of the package. That’s a community discussion we ought to have: do we, as a community, want first-class support in Cabal-the-framework and presumably Cabal-the-library for local package provisioning (via Nix or otherwise)? If so, how?

Feel free to contribute!