Managing a small VPS with NixOS

1. Creating a config
2. Setting up a server (nixos-infect)
3. Deploying to the server
4. Managing secrets
1. Create a master key
2. Add agenix-rekey
3. Encrypt a secret file
4. Use the secret in the configuration
5. Testing on a VM
6. Conclusion

Around 5 years ago, I set up my first virtual server as a place to run some of my Discord bots and other projects. I went into it knowing only very little about system administration and using Linux, but soon after, Linux became my operating system of choice for everyday activities. My interaction with the server was pretty infrequent after the initial setup – I pretty much only logged in whenever there was a new thing I wanted to add to the list of software running on it, or if I had to troubleshoot something. I ran almost all the software on it through Docker compose, and for applications that needed to accept incoming connections from the Internet, I copied and pasted some nginx config and ran certbot for TLS.

Over the years, the server became kind of a mess. Lots of unused and outdated applications lying around, inconsistent setups, no proper secret management and multiple users with flimsy access rights I barely remember. Around half a year ago, after I had already been evangelised by Nix people and especially after reading NixOS in Production, I figured that setting up a server "from scratch" with NixOS could be a great way to learn more about Nix and get a cleaner, more maintainable setup. Here is what I wanted:

• the entire server setup should be a NixOS configuration
• all of the applications running on the server (as well as any system settings) should be defined via NixOS modules
• I should be able to run the exact same setup in a testing environment without much additional effort
• the config should sit in a public Git repository
• deploying the configuration should only require one command and no manual SSH intervention

I lost sight of this project because of my thesis, but now that I'm done with that, I returned to it and managed to whip up something I'm happy with. What follows is essentially a tutorial if you want to follow in my footsteps, i.e. you think the points above sound nice and you are a hobbyist with a couple servers at most who doesn't want to overcomplicate things.

If you do follow along, please tell me in the comments or through other means if you run into something that doesn't work. I may have missed something while breaking down my personal config (which is the basis for all the code shown here).

Creating a config

First, we gotta start with something. For my setup I'm using flakes which is not strictly necessary but well-suited for "standalone" things.

nix flake init

Here's my suggested skeleton for the flake.nix:

{ 
  inputs = {
    # Useful functions for flakes
    flake-utils.url = "github:numtide/flake-utils/v1.0.0";
    # Using a stable version here, you could also use unstable
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.11";
  };

  outputs = { self, flake-utils, nixpkgs, ... }:
    let
      # Function that builds a NixOS system
      vps = { system, name, modules ? [] }: nixpkgs.lib.nixosSystem {
        inherit system;
        modules = [
          # Module (in other file) that defines the system configuration 
          ./vps.nix
          # Module that sets the system host name to the argument (overriding any other defaults)
          { networking.hostName = nixpkgs.lib.mkForce name; }
        ] ++ modules;
      };
    in
    {
      nixosConfigurations.vps = vps {
        system = "x86_64-linux";
        name = "my-vps";
      };
    };
}

Basically, what this sets up is a function vps that produces a NixOS configuration for the server. It includes some default modules (in this case ./vps.nix, which we still have to write) and sets the host name. We use this function in the nixosConfigurations.vps output of the flake, which will be the configuration for the real server, i.e. the one that we'll deploy to the VPS. In my case I already knew that the server was going to be a standard x64 one; if you want to use an ARM server, you have to replace x86_64 with aarch64.

Now it's up to you how to configure your server and the software that should run on it. In vps.nix, you probably want to add some very basic stuff like setting the timezone or opening the firewall for incoming traffic over HTTP(S). I also used it to enable nginx.

# vps.nix
{ config, lib, pkgs, ... }:
{
  services = {
    openssh = {
      # Only SSH key authentication
      settings.PasswordAuthentication = false;
    };
    
    nginx.enable = true;
  };
  
  users.users.root.openssh.authorizedKeys.keys = [ "your-ssh-public-key-1" "your-ssh-public-key-2" ];
  
  networking.firewall.allowedTCPPorts = [ 80 443 ];
  time.timeZone = "Europe/Berlin";
}

Note that I'm not enable-ing the OpenSSH service here. This will be a necessary step later when bootstrapping the actual server. It's a chicken-egg type problem, because we already need some sort of SSH configuration on the server prior to deploying this module for the first time (otherwise we can't connect to the server).

As for the rest of the configuration, you're free to split it up however you want. For my config, I created a directory services with one module per service that should run on the server. I also created some helper options for the common pattern of hosting some web application and configuring nginx to forward requests on a specific domain to it. Feel free to take inspiration from this: here's the helper option declaration, and here is an example of it being used for a web app.

If you want to follow along without actually installing anything specific yet, just add the following static page config to vps.nix.

services.nginx.virtualHosts.localhost.locations."/" = {
  index = "index.html";
  root = pkgs.writeTextDir "index.html" ''
    <html>
      <body>
        Hello, world!
      </body>
    </html>
  '';
};

Setting up a server (nixos-infect)

Ok, now for the fun part. You need a server for this step, preferably one with a freshly installed Linux distribution and no important data on it left. The way we're going to install NixOS on it will wipe its root file system, so proceed at your own discretion. For reference, I run my stuff on the cheapest Contabo VPS, which is a pretty good deal for Europeans, and a slightly worse deal for people in other regions of the world. If you're still looking for a provider, it might make sense to pick one from the list of providers on which nixos-infect has been confirmed to work.

nixos-infect is a shell script that is arguably not the most stable, but certainly a very easy method to install NixOS on a server that's currently running a different Linux distribution. Its usage instructions practically speak for themselves, but I'll rephrase them a little bit:

1. Run your server with some Linux distribution installed. It doesn't really matter which one since it's going to be replaced anyway. I picked Ubuntu 20.04 because that was listed as one that works for Contabo.
2. If you have existing data on your server that you need to preserve, create a backup.
3. Make sure you have an SSH key that you can use to log in as root. If you don't, create one and upload it to your server.
4. Log in on the server via SSH as root, then run

   curl https://raw.githubusercontent.com/elitak/nixos-infect/master/nixos-infect | NIX_CHANNEL=nixos-24.11 bash -x
   

   The NIX_CHANNEL can be chosen freely and it's not super relevant since we're overriding the channel-based config with our flake anyway. But it makes sense to use the same version that you use for the nixpkgs flake input.

The script should run on its own and end with a restart of the server. If anything goes wrong, check the hoster notes in the readme. Worst case scenario is you'll have to tell your provider to reinstall Linux on that server (should be possible from the provider's dashboard).

Otherwise, once the server is back up, you should have a NixOS system that you can log in to via your SSH key.

Deploying to the server

The next step to complete your config now is to copy the config generated by nixos-infect (it includes important base configuration) into your config repo. You can do this quite easily with scp:

# Running on your local machine again now
scp -r <vps-host>:/etc/nixos .

This should add configuration.nix and hardware-configuration.nix in the current directory. I put them in a sub directory prod/ to indicate that they're specific to the actual machine I want to deploy to, in contrast to the rest of the modules which work on any machine. You should take a look at these files to understand what's already configured by default.

Now all that remains is to add this configuration to the vps configuration in our flake:

# ...
      nixosConfigurations.vps = vps {
        system = "x86_64-linux";
        name = "my-vps";
        modules = [ ./prod/configuration.nix ];
      };
# ...

To test that your configuration is error-free, you can run nixos-rebuild --flake '.#vps' build.

After this initial setup, deploying the configuration in the flake to the server is a single command¹:

nixos-rebuild --cores 0 --flake '.#vps' --target-host <host> switch

If you added the nginx virtual host to the config earlier, you should be able to visit http://<host> in your browser to see a "Hello, world!" after you've run this command for the first time!

Managing secrets

Lots of applications you run on a server require some sensitive information as part of their config, like an admin password, an API key or similar. You generally don't want to put these in a plain text file, especially on NixOS where anything configured through Nix will land in the /nix/store, which can be freely read by any process and user of the system.

When I looked for a solution, the most convenient I could find was agenix-rekey. It is an extension of the encryption tool age with its Nix integration agenix. The basic procedure is this:

1. Configure one or more master keys for your secrets. These are age keys that you have access to locally.
2. Write any secrets you need as files. It's important that the components that need them accept files for secret configurations. For example, for the RSS reader Miniflux you can define the admin credentials by specifying services.miniflux.adminCredentialsFile.
3. Encrypt your secrets with the master keys.
4. Get the SSH public key of your server and add it to your configuration.
5. Rekey your secrets so the server can decrypt them.

This system essentially allows you to use one set of master keys to work on your secrets but to also encrypt them for multiple different targets without much friction.

Create a master key

First, you need an age identity to encrypt files. If you happen to have a Yubikey, you can store age identities on there. Otherwise, generate one via age-keygen (after installing age or rage). That key should be protected, so you should either store it in a file outside of the repository or protect it with a password. To create a password-protected key, you can run age-keygen | age -p -o master-key.age. That file (master-key.age) is then safe to store in your repository, provided you've chosen a decent password/passphrase. In the following I'll assume you've done just that so that you have a master-key.age at the root of your repo.

Add agenix-rekey

Add agenix-rekey to the flake:

{ 
  inputs = {
    flake-utils.url = "github:numtide/flake-utils/v1.0.0";
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.11";
    agenix.url = "github:ryantm/agenix";
    agenix-rekey.url = "github:oddlama/agenix-rekey";
    agenix-rekey.inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs = { self, flake-utils, nixpkgs, ... }:
    let
      vps = { system, name, pubkey, modules ? [] }: nixpkgs.lib.nixosSystem {
        inherit system;
        modules = [
          ./vps.nix
          {
            # agenix-rekey uses this option to encrypt secrets for the specific machine we're targeting.
            age.rekey.hostPubkey = pubkey;
            networking.hostName = nixpkgs.lib.mkForce name; 
          }
        ] ++ modules;
      };
    in
    # This makes agenix available as an output of _our_ flake
    (flake-utils.lib.eachDefaultSystem (system: { packages.agenix = agenix-rekey.packages.${system}.default; })) 
    //
    {
      nixosConfigurations.vps = vps {
        system = "x86_64-linux";
        name = "my-vps";
        # This must be set to the SSH public key of your server
        pubkey = "FIXME";
      };
      
      # Which configurations to consider
      agenix-rekey = agenix-rekey.configure {
        userFlake = self;
        nixosConfigurations = self.nixosConfigurations;
      };
    };
}

To find your server's SSH key you can use ssh-keyscan <host>. If that returns multiple, just pick one and put it in the place of FIXME.

You also need to tell agenix-rekey how to store secrets and where to find the master key(s). You can configure these options anywhere, I do it in vps.nix.

age.rekey = {
  masterIdentities = [ ./master-key.age ];
  storageMode = "local";
  localStorageDir = ./. + "/secrets/rekeyed/${config.networking.hostName}";
};

Encrypt a secret file

To walk through this, let's take the Miniflux admin credentials file as an example. Admin user name and password are required settings for running Miniflux. When installing the service via Nix, these values can be set in a file that looks like this:

ADMIN_USERNAME=user
ADMIN_PASSWORD=password

So, we first create this file within the project, e.g. as miniflux-admin.env. Then, we run

nix run '.#agenix' -- edit -i miniflux-admin.env secrets/miniflux-admin.env.age

The file with the sensitive data is now encrypted in secrets/miniflux-admin.env.age. Remember to delete the unencrypted version! Alternatively, you can create a new encrypted file and write its contents yourself by omitting the -i option of the edit command.

Use the secret in the configuration

The point of all this is to be able to reference the secret file from our NixOS configuration. For Miniflux, this can be done now:

# vps.nix
{ pkgs, lib, config, ... }:
# ...

# Encrypted master file
age.secrets.minifluxAdmin.rekeyFile = ./secrets/miniflux-admin.env.age;

services.miniflux = {
  enable = true;
  # Referencing the decrypted version of that file, which will be available on the server after building
  adminCredentialsFile = config.age.secrets.minifluxAdmin.path;
};

Whenever you reference a new secret, you should run

nix run '.#agenix' -- rekey -a

to rekey/re-encrypt it for all hosts (so far, only the VPS) before rebuilding. If an age.secrets entry is defined in the config but hasn't been rekeyed for the configuration you're building, nixos-rebuild will fail.

Testing on a VM

Now, for the bit that makes the config as a whole look a lot more complicated than it actually is: setting up a VM to mirror the server config. Ideally, this allows you to test changes to the config locally before deploying.

I'm taking heavy inspiration for this (as well as some bits of code) from Gabriella Gonzalez' NixOS in Production. Go check that out!

If you're using agenix-rekey, the first thing you must do is create an SSH key pair so the secrets can be encrypted for the VM and the VM can decrypt them in turn. Since the VM runs locally, we won't enable SSH connections to it and therefore, we don't get a "free" SSH host key from OpenSSH (no ssh-keyscan). Manually generating a key pair with no passphrase is easy enough though:

ssh-keygen -t ed25519 -f vm/ssh_host_vm

Since this will be used to re-encrypt the other secrets, do not store the private key part in your (public) repo in plain text. The easiest way to go about this is probably to just add vm/ssh_host_vm to .gitignore and copy it to all your local versions of the repository (if you have multiple) out-of-band. Alternatively, you can also use something like git-agecrypt to store the file encrypted in Git, but have unencrypted access in your local file system. The reason you want to protect this private key in the first place is not that the key itself is sensitive (after all, nobody can remotely log in to your VM). Rather, other people will be able to decrypt your other secrets if they have access to it. If your repo isn't public, this is less of an issue. Just know that if the VM private key is known, the secrets rekeyed for your VM can be easily decrypted.

Once you have an SSH key pair for the VM, create a module vm/vm.nix with VM-specific configuration:

{ config, lib, pkgs, modulesPath, ... }:
{
  # Adds qemu VM support
  imports = [ "${modulesPath}/virtualisation/qemu-vm.nix" ];

  # https://github.com/utmapp/UTM/issues/2353
  networking.nameservers = lib.mkIf pkgs.stdenv.isDarwin [ "8.8.8.8" ];

  virtualisation = {
    graphics = false;
    host = { inherit pkgs; };
    # VM port 80 can be accessed on host machine via localhost:8080
    forwardPorts = [
      { from = "host"; guest.port = 80; host.port = 8080; }
    ];
  };

  services = {
    # Automatic login 
    getty.autologinUser = "root";
    # OpenSSH is not needed for VM
    openssh.enable = false;
  };
  
  # The key used to decrypt agenix secrets
  age.identityPaths = [ ./ssh_host_vm ];

  system.stateVersion = "24.11";
}

Then, add VM configuration to the flake:

{ 
  inputs = {
    flake-utils.url = "github:numtide/flake-utils/v1.0.0";
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.11";
    agenix.url = "github:ryantm/agenix";
    agenix-rekey.url = "github:oddlama/agenix-rekey";
    agenix-rekey.inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs = { self, flake-utils, nixpkgs, ... }:
    let
      vps = { system, name, pubkey, modules ? [] }: nixpkgs.lib.nixosSystem {
        inherit system;
        modules = [
          ./vps.nix
          {
            age.rekey.hostPubkey = pubkey;
            networking.hostName = nixpkgs.lib.mkForce name;
          }
        ] ++ modules;
      };
    in
    (flake-utils.lib.eachDefaultSystem (system:
      let
        # NixOS configuration for running a VM on each possible system 
        machine = vps {
          name = "vm";
          system = builtins.replaceStrings [ "darwin" ] [ "linux" ] system;
          modules = [ ./vm/vm.nix ];
          # The public key to use for secret-reencryption is next to the private key in our repo
          pubkey = builtins.readFile ./vm/ssh_host_vm.pub;
        };

        vmScript = nixpkgs.lib.writeShellScript "run-vm.sh" ''
          export NIX_DISK_IMAGE=$(mktemp -u -t nixos.qcow2)

          trap "rm -f $NIX_DISK_IMAGE" EXIT

          ${machine.config.system.build.vm}/bin/run-${machine.config.networking.hostName}-vm
        '';

      in
        {
          # The VM configuration becomes a non-standard flake output machine."${system}" that can be referenced further below 
          inherit machine;

          packages.agenix = agenix-rekey.packages.${system}.default;

          # Configuring this means that the VM will be started when executing `nix run`
          apps.default = {
            type = "app";
            program = "${vmScript}";
          };
        }
    ))
    //
    {
      nixosConfigurations.vps = vps {
        system = "x86_64-linux";
        name = "my-vps";
        # This must be set to the SSH public key of your server
        pubkey = "FIXME";
      };
      
      agenix-rekey = agenix-rekey.configure {
        userFlake = self;
        # The vps configuration output + all the VMs
        nixosConfigurations = nixpkgs.lib.mergeAttrsList
          ([ self.nixosConfigurations ]
           ++ (map (system: { "machine-${system}" = self.machine."${system}"; })
             flake-utils.lib.defaultSystems));
      };
    };
}

Again, the secrets have to be rekeyed now so that the VM can read them as well.

nix run '.#agenix' -- rekey -a

After that's done, you should be able to start a VM with nix run. Once it's booted up, you can access the VM's version of the hello world page from earlier via http://localhost:8080.

Conclusion

Setting up a server completely with NixOS sounded very scary to me in the beginning, especially since my experience with writing Nix configs thus far had been rather limited (I'm still rocking a NixOS installation with a single configuration.nix on my personal computers). It turned out to be... pretty good! Sure, it took me some time to understand some components (especially agenix), but I did not run into any sort of roadblock, really. Hopefully in finishing this project I've come a step closer to really "getting" Nix. Probably the biggest benefit is that it's become a lot easier now to change my server setup, install new software, and to migrate hosting providers. All in all, I'm glad I put in some time and effort into this.