21 November 2019 ||
Nix's infrastructure offers a cache for the build products of derivations contained in Nixpkgs, which greatly accelerates the CI process. Ideally, the CI cache should benefit developers too: no need to build packages yourself if CI already did. However, sharing the CI cache with developers raises security issues.
In this post, we show an approach based on multi-user Nix and post-build hooks that solves these security issues and more.
Let's walk through the standard procedure for setting up such a Nix cache. First, create a signing keypair:
nix-store --generate-binary-cache-key example-nix-cache-1 ./nix-cache-key.sec ./nix-cache-key.pub
The substituter url and the key generated in
nix-cache-key.pub should be
/etc/nix/nix.conf on developer machines and CI nodes, which should
look like this:
substituters = https://cache.nixos.org/ https://s3.amazonaws.com/example-nix-cache/ trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= example-nix-cache-1:1/cKDz3QCCOmwcztD2eV6Coggp6rqc9DGjWv7C0G+rM=
On NixOS, this can be accomplished by setting
nix.binaryCachePublicKeys = [ "example-nix-cache-1:1/cKDz3QCCOmwcztD2eV6Coggp6rqc9DGjWv7C0G+rM=" ]; nix.binaryCaches = [ "https://s3.amazonaws.com/example-nix-cache/" ];
We used Amazon S3 in our example above but Nix can also use other S3-based backends or simple HTTP/WebDAV-like (supporting GET and PUT) endpoints—there's even a PR for Google Cloud Storage (GCS) support. You can also use Cachix.
The following assumes a public S3 Bucket called
For other binary cache backends, instructions will slightly differ.
Now Nix is configured to substitute from our new cache too, but how do we
upload to that cache? Assuming your project contains a
default.nix with an
deps, a naive solution to upload build artifacts would be something
like the following:
for o in $(nix-build --no-out-link -A deps); do nix copy --to s3://example-nix-cache?secret-key=/run/keys/nix-signing-key $o done
nix-build command will build all dependencies and
nix copy for each of them, which will take care of signing and
uploading the whole closure. If these paths already exist on the binary cache,
uploading will be skipped.
But there is a problem. You can't allow untrusted contributors to upload packages to the cache: they could easily replace the uploaded build artifact by another, possibly with a backdoor. This artifact would masquerade as the build product of a legitimate Nix expression, it would be signed, and used by all the developers' machines, or even production systems. This would be a serious security vulnerability.
On the other hand, preventing untrusted contributors to upload—which creates a build artifact—means that many caching opportunities will be lost. That leads to increased load on CI that can seriously impact the cache hit ratio and roundtrip time when iterating over a work in progress.
This vulnerability can happen in one of two ways:
nix copywill happily sign the faulty artifacts and upload them.
We need to harden both of these vulnerabilities so we can restore the ability to cache build artifacts from untrusted contributors. Let's begin.
Multi-User Nix decouples the steps of assembling the build
nix-instantiate), doing the actual build and writing to the store,
and executes these as different users.
This means that untrusted contributors can upload a "build recipe" to a privileged Nix daemon which takes care of running the build as an unprivileged user in a sandboxed context, and of persisting the build output to the local Nix store afterward.
This "build recipe" effectively prevents untrusted contributors from manipulating the local Nix store—assuming there are no user privilege escalation exploits or hash collisions. They simply can't manipulate it directly, and the only way to get something in the store is by uploading a build recipe, with the output hash being done by the daemon.
Multi-user Nix prevents the manipulation of the local Nix store, and isolates
different parts of a
nix-build between different users. However, we still
need to sign and upload the build artifacts to the binary cache so other
machines can make use of it.
It is clear that we should not let an untrusted contributor sign packages, because this would provide a way to sign and copy bad outputs into the remote Nix store. Luckily, since Nix 2.3, there's a way around this issue. With post-build hooks, the Nix daemon can be configured to execute a script after building a derivation.
In order to do so, add to your
post-build-hook = /etc/nix/upload-to-cache.sh
where a trivial implementation of
upload-to-cache.sh could be:
set -eu set -f # disable globbing export IFS=' ' echo "Signing and uploading paths" $OUT_PATHS exec nix copy --to 's3://example-nix-cache?secret-key=/run/keys/nix-signing-key' $OUT_PATHS
In a multi-user Nix setting, this runs in the context of the daemon (that is, as a root user), so we can properly shield the signing keys from the untrusted contributor, by making sure that the key file if only readable by the root user. In a cloud environment, special care needs to be taken in how the instance retrieves secrets on startup—a regular user shouldn't be able to extract the secret by querying the metadata server for its startup script.
We can now safely upload build artifacts from untrusted contributors to the cache, greatly improving caching. But we gain more along the way.
With the standard setup, the uploading process is a step in your CI pipeline with an explicit list of all the build artifacts you want to upload. This approach has three unfortunate consequences:
depslist fails to build, intermediate build artifacts won't be uploaded:
nix copyis never issued on the store path introducing that dependency, as it was never built.
depssucceed, how do you know you have cached everything? Build dependencies might not all be readily available at a central location that can be called via
nix-build. Often, multiple scripts are invoking Nix by themselves, via
nix-shell, via Bazel rules (rules_haskell, rules_nixpkgs, or calling
nix-buildthemselves during build) to provide dependencies. Manually tracing these dependencies in a
.nixfile (and keeping them in sync) is an arduous task, it gets even harder if some of your packages rely on IFD.
Nix's post-build hook moves the caching logic out of the pipeline definition and into the builder environment—that means it's shareable across all of your pipelines. And, we upload every single build artifact: it doesn't matter if later builds fail, it doesn't matter if we don't know what these artifacts are.
We learned how Nix's new post-build hooks feature can automatically sign and upload artifacts to a binary cache in a trusted way.
How? This post-build-hook setup lets untrusted contributors' artifacts be uploaded on CI: contributors don't have direct access to the local Nix store anymore and don't have access to the signing key. So there's no way to produce a modified signed artifact under the original store path or to upload it to the cache. In some cloud environments, it might still be possible to alter files in the cache. But, since the signatures will be broken, substitutions on other machines using that cache will fail, and they will fall back to building locally.
The post-build-hook setup has other benefits valuable enough that you should consider using it even if you don't have any untrusted contributors. For example, CI pipeline setup is much easier since your developers won't need to remember the invocations required to upload cache, or figure out what list of artifacts they want to cache: it's all configured in the builder already and done automatically. The caching is even improved slightly with successful partial builds since intermediate builds are cached.
Currently, this setup is most suitable when you provide self-hosted builders because:
Multi-User Nix requires multiple users, ruling out some of the hosted CI solutions which don't allow to provide self-hosted runners. However, by just using post-build hooks in a single-user Nix setting, you still have simpler and better caching.
Running Nix inside Docker currently requires a privileged container (because
we want sandboxing to be enabled to ensure some of the isolation properties)
, making it unsuitable for some container platforms. On top of that,
nixos/nix docker image doesn't provide a multi-user Nix
installation, it's a single-user installation based on Alpine (there's
For more on this subject, you can also watch my NixCon 2019 talk.