Rewriting the NixPak launcher in Rust

The NixPak launcher is the small native binary that sits between a Nix-generated wrapper script and bwrap. It reads no config file of its own. Everything arrives through environment variables: NIXPAK_APP_EXE, BUBBLEWRAP_ARGS, BWRAP_EXE, and a handful of presence-means-enabled knobs for the dbus proxy, the Wayland proxy, pasta, and flatpak metadata. It resolves bind mounts, starts helper processes, execs bubblewrap, and then spends the rest of its life reaping children in the right order. The sandbox it sets up is the one described in my write-up of Dolomite; this is the piece that assembles it.

It used to be Go. It is now Rust. The reasons that mattered in the port were not the ones people usually give for a Rust rewrite.

The deleted source is still the specification

When the Go launcher was ported, the Go source was deleted from the tree, and a 490-line reference document describing its exact runtime behaviour was written and deliberately kept. Not as history. As the contract. The doc's own header says: the Go version is the correctness reference; the Rust port must match these semantics exactly.

The reference was reconstructed from git show 0c9c824:lib/nixpak/launcher/main.go and its test file, and it documents things no type system would have preserved:

• xdg-dbus-proxy exits only on G_IO_HUP from the launcher closing its own read end of the sync pipe, so Close() must close the read end before Cmd.Wait() or teardown hangs forever.
• instanceId is md5(pid) rendered in base32 with the custom alphabet 0123456789abcdfghijklmnpqrsvwxyz and no padding, because launch.nix constructs the same name independently and the two must agree.
• resolveSymlinkChain has a depth cap of 40 and returns silently on any lstat error, so a circular symlink yields zero binds instead of a panic.

None of that is expressible as a Rust signature, and every line of it is behaviour the port has to reproduce exactly.

The bug the port reintroduced

The reference document exists because the port broke something first.

Go's os.Pipe() calls pipe2(2) with O_CLOEXEC on both ends, for free, without the programmer ever thinking about it. The launcher's info-fd handshake depends on that: it hands bubblewrap the write end of a pipe at fd 3, closes its own copy after Start(), and then reads to EOF. Bubblewrap writes its {"child-pid": N, ...} JSON, closes the fd, and blocks on fd 4 waiting for permission to execvp the app. Reading to EOF is how the launcher learns both the child PID and that bwrap has reached the block point.

Drop O_CLOEXEC and the write end leaks across exec into some other child. EOF never arrives. The read blocks forever. The symptom is silence: the app doesn't start, with no output and a launcher that never exits.

The Rust port did exactly this. That is why the guide's §9 is a numbered checklist of fifteen invariants with a concrete failure mode attached to each, and why the file carries a DO NOT DELETE THIS DOC banner. Rewriting in a memory-safe language buys you nothing against fd lifetime bugs. OwnedFd will not tell you that a descriptor's survival is the hazard.

What Rust bought

Not safety. Structure: specifically, a place to put alternative bind-resolution strategies.

The whole reason the launcher is complicated is bindEntireStore = false. Rather than mounting the entire /nix/store read-only, the launcher discovers exactly which store paths an app reaches through symlinks and binds only those. For a buildEnv symlink farm like /run/current-system/sw/share/icons that means tens of thousands of lstat/readlink/readdir calls per launch. The firefox fixture walks roughly 68,000 directory entries before the app has drawn a pixel.

The Rust launcher factors this behind a BindResolver trait and an ordered fallback chain, configured by NIXPAK_RESOLVE_STRATEGY as a comma-separated list. Three strategies exist:

Strategy	Mechanism	Bind set vs. walk
walk	the original symlink walker, pure filesystem	baseline
prune	same walk, two provably-redundant syscall classes removed	byte-identical, including order
closure	nix-store -q --requisites, one batched invocation	a strict superset

prune is the current default with walk behind it. It drops the per-entry lstat in favour of the d_type that getdents64 already returned, and drops a canonicalize used only as a memoization key, provably a no-op, because every scan_dir input is canonical by induction.

What prune deliberately does not do is more instructive. It does not skip descending into the subtree of a store root it has already bound. That looks like the obvious win and it is unsafe: a file inside store root X can be a symlink to a different root Y (icon-theme Inherits= chains, library wrapper shims), and binding X does nothing for Y. The only way to find Y is to read X. A blanket prune silently narrows the sandbox until the app breaks. No descent-level prune is provably bind-preserving from filesystem information alone, so the strategy restricts itself to the two reductions that are.

closure trades in the other direction: far less filesystem work, but Nix records all references of a path, including ones a tree-walk would never reach, so the sandbox sees a wider read-only store view. It also declines when nix-store is unavailable, which is what the fallback chain is for.

The wall

closure is opt-in, and the comment in strategy.rs says why: it blows the bwrap argv past ARG_MAX.

Each discovered path becomes a --ro-bind P P triple. Command::spawn ends in execve, and the kernel caps the combined size of argv plus envp. The budget is roughly sysconf(_SC_ARG_MAX) minus the environment, which is why a fat environment lowers the ceiling. Past it, E2BIG, and a failure message that says nothing about bind mounts.

The fix is bwrap --args FD: bubblewrap reads NUL-separated tokens from a file descriptor, parsed inline through the same option loop, so the binds, the --, the app exe and its arguments can all live in the fd and the real command line collapses to two tokens. The fd is a memfd_create, not a pipe: a pipe's 64 KiB buffer deadlocks when the parent writes a blob larger than that before the child reads, and the launcher is deliberately single-threaded, so a background writer thread is not on the table.

Except --args is not the fix, and this is the finding that matters. Reading bubblewrap's source turns up:

static const int32_t MAX_ARGS = 9000;
if (*total_parsed_argc_p > MAX_ARGS)
  die ("Exceeded maximum number of arguments %u", MAX_ARGS);

The counter increments for every token including those read from the --args fd. Bubblewrap refuses more than 9000 tokens regardless of how they arrive. At three tokens per bind that is about 3000 binds, and the closure of /run/current-system/sw routinely exceeds it. --args removes the kernel's ceiling and leaves bubblewrap's, and the second error is just as fatal as the first.

So the design is a ladder rather than a fix. Deliver via memfd; if the token count would exceed bubblewrap's budget, drop closure from the chain and re-resolve with prune; if that still doesn't fit, collapse every store-path bind into a single --ro-bind /nix/store /nix/store, with a loud warning, because that is a real isolation loss and it must never happen quietly; and if even that fails, error out with the token count, the byte count and both budgets named, never a bare E2BIG. Isolation is preserved as long as possible and then traded explicitly.

The rewrite I deferred

The obvious next thought is: reimplement bubblewrap in Rust and the 9000-token wall disappears, because binds become a Vec in memory rather than tokens on a command line. It would also let the launcher and the sandbox fuse into one process, deleting the fd handshakes that caused the deadlock above.

I assessed it and queued it behind everything else. The scope you would own is the namespace creation, the uid/gid map ordering, PR_SET_NO_NEW_PRIVS, the capability drops, the seccomp filter install, and (the dangerous one) the mount setup with its MS_SLAVE|MS_REC propagation and double pivot_root. A bug in the arg parser is a crash. A bug in the mount dance is a sandbox escape, or a silent loss of read-only enforcement, which is worse because nothing tells you. Bubblewrap is small, audited, widely deployed, and receives CVE fixes I would otherwise be writing myself. There is no maintained drop-in Rust reimplementation; the closest real prior art is hakoniwa, which has the right primitives and a programmatic bind API but not bubblewrap's pedigree.

Correctness had to come first, so the fallback ladder ships and the rewrite waits its turn. Two things move it back up the queue: the fallback ladder collapsing to bindEntireStore routinely in practice, or the live mount-injection work turning out to require owning the mount namespace outright. Either one and the rewrite stops being optional. Until then the ladder does the job and the rewrite stays queued behind it.

The threat model is a paragraph. The mechanism is a year of finding out which of your assumptions the kernel and a 3000-line C program do not share.