Every Box I Haven't Logged Into Is Running a Month-Old Config

I Love My NixOS Fleet, But Half of It Is Stale

My fleet has always been deployed pull-style: SSH into the box, git pull, nh os switch. It works. It's also why a box I haven't logged into in a month is a box running a month-old config.

Nothing is broken. That's the problem. Nothing is broken, so I don't log in, so nothing gets updated, so the drift compounds quietly until I'm debugging a host that's three months behind the config I think I'm running.

This is a trial of deploy-rs wired into an existing multi-host flake, so any host updates with one command from my desk. The boilerplate took twenty minutes. The useful result is deliberately boring: choose a host, run one command, and let the configuration already in the flake do the rest.

What deploy-rs Actually Does

deploy-rs is a push deployer. My machine evaluates the target's nixosConfigurations entry, builds or substitutes the closure, copies it over SSH, and activates it. The target stays passive: sshd and Nix, with no agent or git checkout on the box.

Two safety nets ship enabled by default:

• autoRollback: if the activation script fails, the previous generation is restored.
• magicRollback: after activation, the target waits for the deployer to confirm it can still reach the box. If that confirmation never arrives (say the deploy changed a firewall rule or the SSH port and locked you out) the target reverts itself after about 30 seconds.

magicRollback is the feature that made the trial worthwhile. Most of my hosts are only reachable over Tailscale, and the scariest class of change is exactly the one that breaks the path I'd use to fix it. I still need to see it recover a test host before I trust it.

Keep the Node Config Out of the Flake

Three touches to the flake, one new file. The input first:

deploy-rs = {
  url = "github:serokell/deploy-rs";
  inputs.nixpkgs.follows = "nixpkgs";
};

nixpkgs.follows matters here. Without it, deploy-rs drags in its own nixpkgs and your closures drift from what the rest of the flake pins.

The upstream README shows deploy.nodes defined inline in flake.nix. My flake is already 850 lines; the last thing it needs is per-host deployment policy sitting next to the input list. So the nodes live in a separate deploy.nix at the repo root, imported in one line:

deploy = import ./deploy.nix {inherit self deploy-rs system;};

The module itself stays small, because the per-node config is genuinely uniform:

{
  self,
  deploy-rs,
  system,
}: let
  mkNode = name: {
    hostname = name; # Tailscale MagicDNS short name
    sshUser = "justin";
    profiles.system = {
      user = "root";
      path = deploy-rs.lib.${system}.activate.nixos self.nixosConfigurations.${name};
    };
  };
in {
  nodes = {
    nixbase = mkNode "nixbase";
    nixexit = mkNode "nixexit";
  };
}

Three things hiding in that snippet:

• hostname = name works because Tailscale MagicDNS resolves bare hostnames anywhere on the tailnet. No IPs, no FQDNs, no /etc/hosts.
• sshUser is my normal user; user = "root" means deploy-rs escalates with sudo for activation. That only works non-interactively because both trial hosts import my server hardening module, which sets security.sudo.wheelNeedsPassword = false. A host without that needs interactiveSudo = true on its node, which prompts mid-deploy.
• The node name doubles as the nixosConfigurations key, so mkNode physically can't point a hostname at the wrong config.

Pin the CLI in the Dev Shell

deploy-rs has a version-coupling quirk worth knowing before it bites you: the deploy CLI binary and the library the flake uses to build activation scripts must match.

Run a globally installed deploy against a flake pinning a different version and you get cryptic argument-parsing errors. The classic:

found argument '--profile-user' which wasn't expected

The fix is to never install it globally. Expose it through the dev shell instead:

buildInputs = [
  deploy-rs.packages.${system}.deploy-rs
  # ...
];

Now nix develop -c deploy .#nixbase uses the CLI matching the pinned library, and that mismatch disappears from the normal workflow.

The Check That Wanted to Build My Whole Fleet

Upstream recommends wiring deploy-rs's checks into the flake:

checks = builtins.mapAttrs (system: deployLib:
  deployLib.deployChecks self.deploy) deploy-rs.lib;

I did the equivalent, then stopped and looked at what deployChecks actually returns. It's two checks, and they are wildly different animals:

• deploy-schema: a cheap, eval-only validation of the structure of deploy.nodes.
• deploy-activate: a derivation depending on every profile path of every node. Running nix flake check now means building the complete system closure of every host in deploy.nodes, on whatever machine ran the check.

For two servers carrying nvidia drivers, ollama, Jellyfin, and an arr stack, that's not a sanity check. That's "build the fleet on my laptop."

It's also redundant: deploy builds the target's closure at deploy time anyway, so the activate check only front-loads that cost for hosts I'm not even deploying to right now.

So the flake keeps the cheap check and drops the expensive one:

// (builtins.removeAttrs
  (deploy-rs.lib.${system}.deployChecks self.deploy)
  ["deploy-activate"]);

The lesson generalizes past deploy-rs: a bundle of flake checks can combine two very different costs. Eval-time validation is cheap; build-time verification can be arbitrarily heavy. When a library hands you a bundle of checks, look at what each one depends on before merging the whole thing.

Using It

nix develop            # get the pinned deploy CLI
deploy .#nixbase       # one host
deploy .#nixexit       # another host

A dry run first is worth it on a new setup:

deploy --dry-activate .#nixbase      # copy + prepare, don't switch

Flag placement matters: arguments after -- get forwarded to the underlying nix invocation, so --dry-activate has to come before the target.

What deploy-rs Is Not Doing Here

Worth being explicit about the boundaries of this trial:

• It doesn't replace nh os switch on the machine I'm sitting at. Local rebuilds stay local; deploy-rs is for the boxes I'd otherwise SSH into.
• It isn't pull-based CI/CD. Something like comin (which I've run before) has the host poll git and rebuild itself. deploy-rs is the opposite trade: deploys happen exactly when I say, from my machine, with my SSH key, and nothing happens when I don't.
• It doesn't provision. The target must already be a NixOS host I can SSH into. First install is still the installer ISO's job.

Open Questions Before the Rest of the Fleet Joins

• The k3s worker nodes are the obvious next targets, but they need the same hardening module first so passwordless sudo holds.
• remoteBuild = true per node would flip the build onto the target instead of the deployer. Potentially interesting for hosts beefier than the deploying laptop, useless for the ones that aren't.
• magicRollback needs a real test: push a deliberately broken firewall rule at a test host and watch it revert.

That last one isn't optional. Trusting a rollback mechanism I've never seen fire is how rollback mechanisms turn out to be decorative.

That is the shape of the trial so far: deploy-rs replaced a little ritual on every remote machine with deploy .#hostname at my desk. The configuration stays in the flake, the matching CLI stays in its dev shell, and the host no longer has to wait for me to remember to log in.

Header photo by Andrea Lightfoot on Unsplash.