Towards a simple continuous delivery mechanism for NixOS

Here's a simple question that most developers have asked themselves before: how do I get the feature/fix I just pushed to appear in the live application that runs on a server somewhere? If you already have a server and know how to operate it, this is mostly a question of convenience — you could always just run a handful of commands by yourself after every update — but it is a valid question.

One of the most common answers to this question is the use of a push-based mechanism: your update of the application's Git repository triggers some script that updates your server. This can sometimes be as simple as rsyncing a bunch of PHP files over via SFTP, but it can also be a source of cumbersome ceremony if there are a bunch of virtualisation layers between what you're running and where you're running it (think building a Docker image and deploying it to a Kubernetes cluster). What the push-based mechanisms have in common is that they require CI if you want things to happen automatically. And despite having written CI pipelines for a living for over 2 years, I'm not known to be the biggest evangelist of the concept.

Where there's a push-way, there's also a pull-way. Instead of having a CI script update the state of your server imperatively, you have the server monitor the state of your repository. When the remote machine sees that a change has been made, it updates itself. Personally, I generally favour this approach over the other one, mainly because I find a clean separation between application code and infrastructure/deployment logic to be desirable. I prefer it when the repository for the random web app I want to run doesn't have credentials to nuke my server.

But also — and you may have guessed this already — pull, in principle, doesn't need CI infrastructure besides the server itself. In practice, there are obviously still a bunch of good reasons for why you would want CI, especially when building your application is expensive and you deploy to something like Kubernetes, where the base assumption is that everything you want to run has already been packaged as a Docker container.

The starting point

I recently started a small side project, a classic web application written in Clojure. Previously, I've written about how to package Clojure applications with Nix so I can install them on my NixOS server to begin with, but I was still left with the initial question of this post.

When you want to deploy your application as a NixOS module, you are met with some challenges that you won't encounter with other operating systems. Notably, you can't just update one piece of your NixOS configuration independently, you have to rebuild the entire system every time you want to deploy a new version of your application. Additionally, NixOS is pretty strict about reproducibility – it's not possible to include a "dynamic reference" to a module in your configuration, i.e. one that pulls the most recent version of your application when you rebuild.

So, is it impossible to have a NixOS configuration that automatically pulls and runs the latest version of an application?

NixOS containers to the rescue?

Fortunately, NixOS has a mechanism to partition your system configuration into a main configuration and "sub-configurations": NixOS containers. These containers are largely isolated from the main system, but not as virtualised as something like a Docker container. You configure them basically as if you were configuring a second machine, just that it ends up living inside a host system.

The main reason this feature caught my eye was the simple fact that you can configure a container to fetch its configuration from a remote Git repository, without locking it to a specific revision. So the idea was simple: define a container that runs the latest version of my application in the repository of my application — this would be easy since I had already set up a Nix flake for building it — and then reference the main branch of that Git repository from my server configuration.

Combine that with the NixOS autoUpgrade feature, and we're good to go, right? Well,

...autoUpgrade doesn't work in containers

Yeah. That.

I was (naïvely) assuming that pretty much every NixOS feature would work inside NixOS containers, including system.autoUpgrade. But checking the logs of the upgrade job, I was greeted with this:

Feb 06 09:10:00 nixos systemd[1]: Starting NixOS Upgrade...
Feb 06 09:10:00 nixos nixos-upgrade-start[2823]: error: opening lock file '/nix/var/nix/db/big-lock': Read-only file system

As it turns out, NixOS containers share a bunch of files with the host via bind mounts – mostly files under /nix. Which makes sense; you build the container configuration on the host, so all the packages that are required in the container are already present on the host and can simply be reused. But since all those bind mounts are read-only, features like autoUpgrade don't work.

So, autoUpgrade is out of the picture. Luckily, there is another way to update a running container: the nixos-container update command! This command pretty much just rebuilds the NixOS configuration of the container from the outside, like a remote nixos-rebuild switch. Since the container's configuration is given as a remote flake, this should result, in theory, in that flake being downloaded and (re)built. Thus, I can build my own container autoUpgrade by writing a systemd job on the host that runs nixos-container update <name> every couple minutes.

...Right?

...nixos-container update doesn't fetch the latest version

I was already celebrating my genius idea when I realised that nixos-container update sometimes just didn't do anything, even though I had updated the application repository. What happened?

Well, nixos-container unfortunately isn't really well documented. There is an output if you type nixos-container --help, but that just lists all the available subcommands, not how they work. Digging further, I found that nixos-container isn't really a full-fledged tool but rather a single-file Perl script that just calls a sequence of Nix commands for most of its subcommands.

So, what nixos-container update really does for flake-based container definitions is (roughly):

1. call nix build $flake
2. reload the systemd service responsible for running the container

The problem lies with 1.: nix build has a download cache for remote flakes that is invalidated regularly, but not necessarily every time you run it, meaning on most invocations it will not re-download a flake even though it may have changed.

To get around this behaviour, nix build has an option --refresh that makes it "consider all previously downloaded files out-of-date".

Now it was just a matter of getting nixos-container to pass this flag along to nix build, which it wasn't able to when I reached that point. One nixpkgs PR later, and that problem disappeared as well!

And what about secrets?

With a NixOS configuration in your source repository and a container definition + update job in your server config you already have a working CD mechanism. What's still missing is a discussion about how to handle secrets.

Running most applications will involve configuring a password, an API key or something similar. Since the application config is part of the container, which in turn is configured in a source repository, you shouldn't just put sensitive data in there in plain text (that is never recommended for NixOS configurations anyway).

Personally, I'm a fan of agenix-rekey, which makes using agenix a bit more ergonomic. I've described my setup in my previous NixOS post.

The NixOS wiki suggests a way to use agenix by mounting the host SSH key into the container and then having agenix decrypt any required secrets into the container directly. This is a pretty decent approach as far as I'm concerned. The only issue I have with it is that it requires your (encrypted) secrets to reside in the application repository, which is a little annoying in the sense that there isn't a single place to manage all your secrets anymore. So what I ended up doing instead was configuring the secrets on the host (as I would for any other service) and then mounting the resulting files directly into the container. This approach has the additional advantage that it's secret-management-agnostic – all you need is for your secrets to exist as files on the host somewhere.

What you'll probably run into if you mount the secret files directly are permission issues. E.g. with agenix, secrets are only readable by root by default. If you want a non-root user in your container to read the mounted secret, you need to do one of three things:

1. run your application as root (bad, not good)
2. configure the secret to be owned by a specific UID/GID on the host, create a user with the same UID/GID in the container and run your application as that user (annoying, because you are now responsible for keeping magic numbers in sync)
3. define a job that changes the ownership of the mounted files before your application runs (hacky, but probably the cleanest solution)

I opted for the third alternative.

Conclusion

Let's tie this whole thing together, shall we? To summarise the setup, we

• define a NixOS configuration in the application repo, to be used in a container
• define the container settings in the NixOS configuration for the server where the application should run

flake.nix in the application repo:

{
  name = "FIXME";
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
  };

  outputs = { self, nixpkgs, flake-utils, clj-nix }: {
    nixosConfigurations.container = nixpkgs.lib.nixosSystem {
      modules = [
        ({ pkgs, ... }: {
          nixpkgs = {
            hostPlatform = "x86_64-linux";
          };

          boot.isNspawnContainer = true;

          system.stateVersion = "26.05";

          # Depending on how you decide to go about secrets, you may not need this.
          systemd.services.chown-secrets = {
            # This is assuming your application runs as a systemd service with that name
            wantedBy = [ "${self.name}.service" ];
            script = ''
              chown -R ${self.name}:${self.name} /run/secrets
            '';
            serviceConfig = {
              Type = "oneshot";
              RemainAfterExit = "yes";
            };
          };

          users.groups.${self.name} = { };
          users.users.${self.name} = {
            isSystemUser = true;
            group = self.name;
          };
          
          # Configure whatever is needed to run the application in this container
          # Important: it must run as a user named self.name for the secret management stuff above to work. 
        })
      ];
    };
  };
}

NixOS module declaring a NixOS container for your application (part of your server's NixOS config):

{ config, lib, pkgs, ... }:
let name = "FIXME"; 
in {
  systemd.services."container@${name}".path = [ pkgs.git ];

  systemd.services."${name}-update" = {
    script = ''
      nixos-container update --refresh ${name}
    '';
    path = [ pkgs.nixos-container pkgs.nix ];
    startAt = "*:00/5:00"; # every 5 minutes
  };

  containers.${name} = {
    autoStart = true;
    flake = "FIXME"; # URL to the application flake (see https://nix.dev/manual/nix/2.28/command-ref/new-cli/nix3-flake#flake-references) 

    bindMounts."/run/secrets" = {
      hostPath = "/wherever/your/decrypted/secrets/are"
      isReadOnly = false; # must not be read-only for `chown`
    };
  };
}

Overall, this may not be the slickest, most flexible, or most efficient solution, but it is probably one of the easiest way to get changes from an application's Git repo to the live version on your NixOS server without manual intervention or CI. Perhaps in the future this could be turned into a library that you can just add to your flake(s), to reduce friction even further.