Native Linux Builder

Determinate Nix has a feature called Native Linux Builder that enables you to build Linux derivations on macOS using the built-in Virtualization framework rather than needing a remote builder or a Linux virtual machine. This document is a guide to using and handling problems related to the builder.

Gaining access to Native Linux Builder

Many perceived issues with the builder stem from not yet being in a position to use it. The instructions below tell you how to get there.

Install Determinate Nix

The builder is available only in Determinate Nix. See installation instructions here.

You need to sign up for FlakeHub to use Native Linux Builder. Although FlakeHub does have paid features, like private flakes and FlakeHub Cache, the initial signup is free.

Once you’ve signed up, you need to log in to FlakeHub to use the builder. At any time, you can check if you’re logged in using Determinate Nixd:

Check your authentication status with FlakeHub


determinate-nixd status

Obtain access

Native Linux Builder is currently available to a subset of Determinate Nix users. If you’re eager to try it out, contact us to request access and include your FlakeHub user name; if you haven’t yet been granted access, you can’t use it yet.

Addressing common problems

If you run into issues with Native Linux Builder, including error messages indicating that the builder isn’t enabled or available, there are two things you should try, in order:

If you have access to the builder and are logged in to FlakeHub, try logging out and then back in:

Log out of FlakeHub and back in


determinate-nixd auth logout
determinate-nixd auth login

When prompted for a FlakeHub auth token, you can use your existing token:

Copy your existing token to the clipboard


cat /nix/var/determinate/token | pbcopy

Once you’ve done that, try building a Linux derivation again. Here’s an example in case you don’t have one handy:

Build a Linux derivation from Nixpkgs


nix build --print-build-logs --substituters "" \
  "https://flakehub.com/f/DeterminateSystems/minimal-stdenv/0.1#packages.x86_64-linux.default"

Restart Determinate Nixd

If you’re sure that you have access to Native Linux Builder but are still having issues with it, the best thing to do is usually to restart Determinate Nixd using launchctl:


sudo launchctl kickstart -k system/systems.determinate.nix-daemon

Try building a Linux derivation again. Here’s the same example from above in case you don’t have one handy:

Build a Linux derivation from Nixpkgs


nix build --print-build-logs --substituters "" \
  "https://flakehub.com/f/DeterminateSystems/minimal-stdenv/0.1#packages.x86_64-linux.default"

If you’re still stuck, get in touch with us.

Known issues

Out-of-memory related build failures

By default, Native Linux Builder’s VM is spawned with access to up to 8 GiB of system memory.

If you need to build derivations that require large amounts of memory (larger than the 8 GiB default), you can configure the memory limit of the VM in the Determinate Nixd config file by setting builder.memoryBytes to your desired memory size.

Hash mismatches

Native Linux Builder mounts the host Nix store into the VM for the builder to copy the build results to. On macOS, by default, the Nix store is case insensitive; when a build writes files that differ only in case to the store, the contents end up overwritten rather than writing to two distinct files in the store. The Nix case hack does not solve this because that case hack only works on the NAR level—Nix applies it when unpacking the NAR, but does not have any way to apply it when a build is writing directly to the store.

The only known workaround is to have a case-sensitive store volume: you can achieve this by completely uninstalling Nix and then reinstalling it, while directing the installer to create a case-sensitive volume:


# This completely removes your Nix store.
# If you have any processes running from within the Nix store when you do thise, you may experience unpredictable behavior.
/nix/nix-installer uninstall
 
# This reinstalls Determinate Nix and creates the Nix Store volume case sensitive.
curl -fsSL https://install.determinate.systems/nix | sh -s -- install macos --case-sensitive --extra-conf 'use-case-hack = false'

You can verify that this worked by creating a pair of files that differ only by case:


sudo touch /nix/a /nix/A
ls -al /nix
# see that both files exist
sudo rm /nix/a /nix/A

Stalling when building fixed-output derivations

When building derivations that depend on a large number of fixed-output derivations (FODs), you may notice the build stalling. This is caused by the VM needing network access to download the FODs, which we approach by relying on the host machine’s DHCP server. The macOS DHCP server (at least as exposed to the VM) has a fixed lease expiration time of 1 hour, and does not allow us to release the lease when the VM shuts down. That means that if you build 253 FODs within the same one-hour rolling window, the 254th FOD hangs until a lease expires.

If you are not relying on anything that depends on the built-in DHCP server, you may immediately reset these leases by removing the file /var/db/dhcpd_leases. The next FOD build will pick up a brand new lease and should continue working again, until you hit the 253 limit.

”Permission denied” caused by `cp --no-preserve=mode` and similar commands

Native Linux Builder mounts the host Nix store into the VM for the builder to copy the build results to. When using the macOS Virtualization framework, the best way to do this is by using the included virtio filesystem driver. Some derivations may use cp --no-preserve=mode or similar commands when copying outputs into the store. When the destination lives in the host’s Nix store (as is the case when copying final build results to $out, for example), this fails due to a weird interaction with the included virtio driver and how cp --no-preserve=mode works.

The only fix for this is to change the cp --no-preserve=mode call to something else (such as separating this into a cp followed by a chmod). For an example of what this fix would look like, see ipetkov/crane#964.