Nix and NixOS for DevOps

“Works on my machine“ has been the unhelpful answer when the CI or, even worse, production fails catastrophically. Nix, among other things, is an approach to fix this by providing reproducible, declarative, and reliable systems. This makes it a great tool for both sides of what is commonly called DevOps: the development and the process of operating systems. This post will show both with a hands-on example, but first, let’s look at what those promises exactly mean from a birds-eye perspective.

Reproducible

Nix takes inspiration from a functional programming language, modeling the build process of packages and configuration itself as “pure“ functions. The same inputs should always lead to the same outputs. While that sounds rather abstract, in practice this means if it runs on your notebook, it will run on the production server, the CI, or your colleague’s machine exactly the same. Nix strictly prohibits you from using undeclared dependencies, everything is pinned by a cryptographic hash, and packages are evaluated and built in isolation from each other.

Declarative

While a simple script or other more complex tools like Ansible will tell your machine “what to do“ to get to a certain state, when writing Nix code you will be “defining what something is“. This is not a new concept for anyone who has done any functional programming. But while the difference is small, it is of big consequences. Declarative configuration is a big part of the reproducibility discussed above and the idempotency of deployment processes. Nix allows not only to share configuration and build instructions, but also the development and build environments for a project itself. With a few lines of code, you will be able to add a development environment with all the tools, programming languages, and pinned versions that every developer in the team will need to work on a project.

Reliable

The installation of a package should never break a different one. The same goes for upgrades between versions of the same software. Nix not only provides atomic upgrades but ensures that one package will never be able to break a different one. You even can (and often will) have multiple versions or configurations of the same package coexisting happily on the same system. Even better, you can roll back or switch between what is referred to as generations as you wish. During an upgrade, there will never be an inconsistent state, imagine installing and configuring the new version of the package alongside the old one, and when everything is ready, switching to it.

OK, how does it work?

First up, a bit of terminology to avoid confusion. Nix refers to two things:

• The declarative package manager
• The programming language used to define packages and configuration

NixOS on the other hand is a Linux distribution built around the Nix package manager, which uses its capabilities to define the whole system configuration.

Nix breaks with the commonly known Linux file system hierarchy. If you ever have used it, you will have stumbled upon a pretty big directory under /nix/store with a lot of cryptically named paths in it. This is the so-called Nix store, where Nix stores all packages in its own unique subdirectory, such as:

Throwing everything into one big directory might sound counter-intuitive but this is actually the concept that enables most of the magic of Nix. Every store path has the same structure:

Where the hash is a unique identifier calculated from all the package’s dependencies (to be precise: a cryptographic hash of the package’s build dependency graph).

But… why?

This very simple structure allows for a lot of desirable features. First of all, the same path will have the exact same contents. This is our first step in ensuring reproducibility. It also ensures the files will not have been tampered with without you knowing, as the hash verifies the content mathematically.

You can now have multiple versions of the same package all at once. They will have different hashes and thus lie in different paths in the store. Instead of dependency version conflicts between packages (i.e. “DLL-hell“), every package can reference and use its own desired version.

An important consequence is that operations like upgrading or uninstalling an application cannot break other applications, since these operations never “destructively“ update or delete files that are used by other packages. This enables us to have what is referred to as atomic upgrades and rollbacks.

Getting rid of global locations like /usr/lib on a traditional Linux system ensures no hidden dependencies that are not specified in the package. If it builds on one system, it will build on a different one regardless of what other software is installed.

In practice, packages are built from Nix (language) expressions. From here, Nix computes a derivation. This is best described as a “build action“, a complete specification of all dependencies, tools, environment variables, sources, and steps needed to build a certain package. Derivations are independent of programming languages and deterministic.

Having every package and part of your system strictly identified by a hash also allows for powerful caching. When instructed to build

from the source, Nix would first check if it already exists in the store or any of the configured caches and pull it from there if already present. In practice, building from source becomes a mere fallback for commonly used packages.

Nix flakes

Flakes are a newer feature of Nix, best described as “a mechanism to package Nix expressions into composable entities“. In practice, a flake is a file system tree (typically fetched from a Git repository or a tarball) that contains a file named flake.nix in the root directory.

They are a standardized interface to split and distribute Nix code and make reuse and composition easy. We will use this format for the following sections and build your flake.nix file step by step further explaining on the way.

A practical example

With all that out of the way, let’s see how Nix is used in practice. The following example will demonstrate the DevOps process of building (the “Dev“ in DevOps) and deploying (the “Ops“ in DevOps) an example Go application using Nix everywhere we can.

The Development (Dev)

For the “Dev“ part, let’s first present our example application. A very simple web server written in go. It is assumed we are using Go modules, i.e., have run go mod init and go mod tidy to create a go.mod and go.sum file for our application.package main

In its simplest form, a flake is an attribute set that defines the attribute’s outputs and (optionally) inputs and description. Inputs are the sources or dependencies used in the flake, in most cases git repositories or plain URLs to fetch from. Outputs are commonly packages, deployment instructions, configuration or Nix code (functions) to be used in other flakes. While the outputs attribute set can have any form you want, there are standardized names for common outputs.

To “nixify“ our project, let’s start by adding a file called flake.nix to the root of the repository, with a very basic flake:

Our flake defines github.com/nixos/nixpkgs as an input and has no outputs for now. While it is not very useful yet, it is functional. We can use nix flake metadata to display some information about it.

If not already present, this will also create a flake.lock file for us, pinning all our inputs to a specific version, verified by a hash of the input.

Adding a package

With the basic boilerplate done, we can add our first output: A package for our application.

The above specifies how our application is to be built on Linux systems with x64_64 system architecture. We could use the generic mkDerivation function to define our package (derivation), but Nix includes helper functions for most common programming languages and frameworks. Here buildGoModule is used to define the package for our Golang project, specifying the source to be the current directory ./., a name, version, and the hash of the dependencies specified in our go.mod file, since all versions have to be pinned with a hash.

We can now use nix build to build our application. This will build our application and create a symbolic link called result pointing to it in the nix store.

Okay, but we could have saved ourselves the hassle and just used go build to get the same thing, right? Well, not quite.

Nix and especially flakes promise us a hermetic evaluation, meaning everything needed to build our application is pinned to a specific version and verified using a checksum. This includes the code of the app itself, but also the compiler used, any other dependencies, tools, and packages used, and the system architecture itself. Not only does that ensure our app will not break because of external factors, but it will also build on any machine. Be it a coworker’s notebook, CI, or production, there are no more “works (only) on my machine“. Since Nix is language-agnostic and not specific to go, we also have a common interface for building software in any language. The process for building and running (nix run) a nixified go application stays the same.

It should be noted explicitly that none of the prerequisites like a go compiler or any other tools (apart from Nix itself) need to be installed on the system before we can build or run the application with Nix. Nix will create the complete environment specified by the flake. Inputs will be fetched on the first run and subsequently checked from there on with the hash to ensure a deterministic build while caching as much as possible.

Adding a shell (Ad-hoc environments)

While the package definition already allows us to build and run our application, it is not really suited for development. By adding a Nix shell to the outputs we can create a development environment with all the tools in their specific versions that all developers can use. Instead of every developer wasting time on setting up a tech stack for a new project, we can pass around a shell with all tools needed to work on it available in their specified version.

Since the requirements for our project are all already packaged in nixpkgs we can add a simple shell with the following lines in our outputs:

Now, nix flake show will show us two outputs: our previously specified package and the newly added shell.

We can enter the shell by running nix develop and will be dropped into a shell
with all tools we need available.

In this example, we have only used tools and dependencies available prepackaged in nixpkgs. It is of course possible to specify tools outside of nixpkgs or to override specific versions of packages. This can be archived by either adding your own package definitions or using overrides and overlays) to modify the existing ones on the fly. The tools specified in buildInputs of our shell are of course pinned to a version as well since our nixpkgs input is pinned in the flake.lock file to a specific git commit.

The Operation (Ops)

Until now, we have only used Nix, the package manager, which is as stated in the beginning, a separate tool. For the operations part, let’s take a NixOS, the Linux distribution, which takes the Nix principles and applies them to the complete system configuration. The core idea is simple and just a natural extension of what we have seen with Nix already: In the same manner as our packages are stored in isolated paths like /nix/store/5rnfzla9kcx4mj5zdc7nlnv8na1najvg-firefox-3.5.4/ in the store, let’s also apply that to configuration. For example, the configuration for an SSH server could be defined in the Nix language and stored in a path like /nix/store/s2sjbl85xnrc18rl4fhn56irkxqxyk4p-sshd_config after evaluating it.

This results in the complete operating system making use of the benefits we saw with packages: It’s isolated, can be rolled back, and will not change without notice as everything is once again verified by the hash.

The nixos-rebuild tool is used to perform these operations. To switch to a new configuration:

After every build of the configuration, you get what is referred to as a generation. This is the point to which you can switch or roll back to. In the same way, if something goes wrong or does not work as expected

will bring you to the generation before. At boot, you can select any generation you have saved.

To clarify the extent, NixOS is completely built around the Nix package manager. It’s used to build the kernel, applications, system packages, and configuration files in a purely functional language. This results in atomic system upgrades and the ability to roll back the full system, among other things. Furthermore, the system is completely reproducible, which makes re-installation or replication easy. Another example:

will build your system configuration inside a virtual machine and start it. The VM will have the same configuration as your host system itself, allowing you to test, experiment, and “preview“ as you like without risking breaking the host.

Configuration

The system configuration is usually defined in a file called configuration.nix

A very simple example could look like this:

This minimal configuration is enough to build a machine with a running SSH daemon. When using flake, we will also add the configuration to our flake.nix flake under the nixosConfiguration output.

Then pass the flake to the nixos-rebuild command

Modules

All options you are able to set in your system configuration, come from modules which are files containing NixOS expressions. The configuration.nix file is also a module. Most other modules are in the nixpkgs repository. Modules are files that are then combined by NixOS to produce the full system configuration.

It is outside the scope of this post to explain the module system in its entirety, so let’s focus on a practical example for our application from above. We will add a module for it to your flake.nix file so that anyone can just import it, resulting in a running instance of your application.

The module will have only one option called “enable“, which when set to true will add the part under config= to the system’s configuration. This is the full module saved in module.nix:

To import it, just add it to the nixosConfigurations output the same way we added the configuration.nix file:

The module creates a systemd unit running our application. Commonly you would also add configuration for a reverse-proxy here, as well as environment variables, other dependent applications and settings, users, groups, and other properties that should be present when our application is running.

Deployment

Assuming we have a running NixOS host with SSH access, we can deploy your configuration

And voilà, our application is running!

A note about Docker

When introducing to nix, a recurring question is: “What about docker?“ While Nix and Docker are essentially tools for different things, it is unquestionable that people (ab)use both to solve overlapping problems.

Docker’s missing reproducibility

In regard to reproducible environments, Docker images can be used to archive something similar. Note, that specifically build images are meant, as a Dockerfile or container definition is not reproducible. This can be easily confirmed by building an image multiple times and comparing its checksum.

This can be attributed to multiple factors, e.g., Dockerfile commands

What is “latest“ in this expression?

Which version of python will get installed when running this code?

Which of possibly multiple entries in $PATH will get run?

There are workarounds for some of these problems, but Docker (or more specifically Dockerfile) is not built to be reproducible in the first place and there are many real-world examples of image building today but not in a few months‘ time.

Nix + Docker

But the question does not need to be one or the other. In fact, Nix and Docker are used together and can complement each other perfectly.

Nix to build Docker iamges

Nix can be used to build docker images. Nix provides multiple utility functions like pkgs.dockerTools.buildImage and pkgs.dockerTools.buildLayeredImage that can be used to build docker images as flake output, just like any other package. A recent lightning talk about its use was recently given by Matthew Croughan at MCH2022.

It can also be used for more fine-grained optimization for caching of Docker layers as detailed by Graham Christensen.

NixOS to deploy container

NixOS provides a very elegant way of running containers. This is an example of running configuring NixOS to run a container

It will set up everything needed to run the container and generate matching systemd units to start, stop and manage it. All pinned by hashes.

Using Nix and NixOS in combination with container runtimes is a topic too broad to cover in its entirety here and the examples above only are able to scratch the surface of what is possible.

Further reading

This only scratches the surface of what is possible with Nix and NixOS. There are multiple layers of abstraction above and below and tools to simplify the usage and configuration. Deployment tools to manage a large number of hosts like nixOps or lollypops, secrets management like sops-nix or agenix and CI systems like Hydra. Nix plays nicely with containers and its ecosystem is constantly growing and evolving with 2,981 Pull requests merged by 534 people in the last month alone.