OpenTelemetry tracing in Determinate Nix

Determinate Nix can emit OpenTelemetry traces containing spans describing many of the actions that it performs, including evaluation, local and remote builds, cache substitutions, HTTP requests, and Nix daemon connections. It exports those traces over the standard OpenTelemetry Protocol (OTLP), which means that you can send them to any receiver that accepts OTLP over HTTP and inspect them alongside the rest of your telemetry. Such receivers include OpenTelemetry Collector, Jaeger, Grafana Tempo, Honeycomb, and Datadog.

Determinate Nix uses the http/json variant of OTLP: it sends each batch of spans to your endpoint as an HTTP POST with a JSON body and a Content-Type: application/json header, compressed with gzip unless you turn compression off. It doesn’t implement the http/protobuf variant and ignores OTEL_EXPORTER_OTLP_PROTOCOL and OTEL_EXPORTER_OTLP_TRACES_PROTOCOL, so put an OpenTelemetry Collector in front of any backend that accepts Protobuf payloads only.

Tracing is off by default. Determinate Nix never exports traces unless you explicitly enable tracing and configure an OTLP endpoint.

Enabling tracing

You can enable tracing either through your Nix settings or through the standard OpenTelemetry environment variables. We recommend using the Nix settings for long-lived configuration, such as on a build server, and environment variables for things like one-off runs and in CI/CD workflows. If your collector needs an API key, see Authenticating with your collector before you put anything in nix.custom.conf.

Using Nix settings

Add the otlp and otlp-endpoint settings to your nix.custom.conf file:

otlp = true
otlp-endpoint = https://otel.example.org

The endpoint is the base URL of your OTLP collector. Determinate Nix appends /v1/traces to that base URL automatically, so don’t include that path yourself and don’t add a trailing slash (the resulting double slash may confuse some collectors).

The otlp switch enables you to turn tracing off without removing the endpoint from your configuration. If you set otlp = true without an endpoint, Nix warns you and doesn’t export anything.

Determinate Nix supports these four OTel settings:

NixOS

If you’re on NixOS and installed Determinate Nix using the determinate NixOS module, you can set OTel configuration using the standard nix.settings option, which the module writes to nix.custom.conf for you. Here’s an example:

nix.settings = {
  otlp = true;
  otlp-endpoint = "https://otel.example.org";
};

nix-darwin

If you’re on macOS and manage your system with nix-darwin, you can set OTel configuration using the determinateNix.customSettings attribute in the determinate nix-darwin module. Here’s an example:

customSettings = {
  otlp = true;
  otlp-endpoint = "https://otel.example.org";
};

Using environment variables

Determinate Nix honors the standard environment variables for OTLP exporters that are listed below, and ignores the others. Setting an endpoint variable (OTEL_EXPORTER_OTLP_ENDPOINT) enables tracing on its own, even if otlp is false in your Nix settings:

OTEL_EXPORTER_OTLP_ENDPOINT="https://otel.example.org" nix build ".#my-package"

OTEL_EXPORTER_OTLP_ENDPOINT behaves like the otlp-endpoint setting in that Determinate Nix automatically appends /v1/traces to the base URL. If your collector uses a non-standard path, set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT instead, which Determinate Nix uses exactly as you provide it.

The environment variables, listed in the table below, always take precedence over the corresponding Nix settings:

The Determinate Nix manual documents each variable in full.

Authenticating with your collector

Most hosted backends require an API key or bearer token, which you pass as an HTTP header. The otlp-headers setting and the OTEL_EXPORTER_OTLP_HEADERS variable share the same syntax: a comma-separated list of name=value pairs with percent-encoded values. The value authorization=Bearer%20<token>, for example, sends the header authorization: Bearer <token> with every upload. The %20 is the space between Bearer and the token.

Where the token should live depends on which process is exporting:

• Interactive use and CI: set OTEL_EXPORTER_OTLP_HEADERS in the environment, sourced from your CI provider’s secret store or a local secrets manager.

  OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer%20$OTEL_TOKEN" \
    nix build .#my-package

  Referencing $OTEL_TOKEN rather than pasting the token in keeps it out of your shell history.

• The daemon and remote builders: these processes don’t see your shell environment, and environment variables don’t survive the SSH hop to a remote builder. Run an OpenTelemetry Collector on the machine, point Nix at it with no headers at all, and let the collector’s otlphttp exporter hold the credentials in its own root-only configuration file. Nix never sees the token, and you also get batching, retries, and centralized sampling in one place. If you can’t run a collector, put the otlp* settings in a separate file that can only be read by root and pull it in with !include:

  /etc/nix/nix.custom.conf

  !include /etc/nix/otlp.conf

  /etc/nix/otlp.conf (mode 0600)

  otlp = true
  otlp-endpoint = https://api.example.com
  otlp-headers = authorization=Bearer%20<token>

  Nix silently skips an included file that it can’t read, so the daemon picks up the settings while unprivileged clients neither export traces nor see the token.

Compression

Determinate Nix compresses uploads with gzip by default. If your collector doesn’t accept compressed payloads, set otlp-compression to none (or set OTEL_EXPORTER_OTLP_COMPRESSION=none).

Sampling

By default, Determinate Nix records every trace if tracing is enabled. In some settings, like a busy build farm, that can produce loads of data, and in those cases we recommend sampling using the standard OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG variables (neither has a Nix configuration equivalent). To keep 1% of traces, for example:

export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.01

OTEL_TRACES_SAMPLER selects the sampling strategy, which decides whether to record a trace at the moment it starts. Determinate Nix supports the standard samplers:

OTEL_TRACES_SAMPLER_ARG is the parameter for the sampler you chose. It only matters for the two traceidratio samplers, where it’s the fraction of traces to keep, from 0.0 to 1.0; the always_* samplers ignore it. If it’s unset or isn’t a number, Nix warns and keeps every trace. Likewise, if OTEL_TRACES_SAMPLER names a sampler Nix doesn’t recognize, Nix warns and falls back to parentbased_always_on.

The parentbased_* samplers are the ones you want in practice. A single nix build involves several processes: the client, the daemon, and possibly remote builders. The client starts the trace and makes the sampling decision, which then travels to the other processes in the sampled flag of the W3C trace context. A parentbased_* sampler on those processes inherits that decision rather than making its own, so a trace is either recorded as a whole or not at all. The plain traceidratio sampler samples by trace ID for the same reason: every process that sees the same trace ID makes the same decision.

What gets traced

Each Nix invocation produces one trace with a root span named after the command, for example nix build or nix nario, with the service.name resource attribute set to the program name, such as nix or nix-daemon. If the command fails, the root span’s status is set to the error message.

Nix’s activities are represented as child spans under that root, nested the same way they are in Nix’s own progress output. The span types you’ll see most frequently:

The nix.drv.name and nix.store.name attributes carry the package name and nix.drv.version and nix.store.version the version, split out from the store path. Filtering on nix.drv.name = "openssl", for example, is a lot easier than pattern matching on hashes.

A failed build or substitution marks its span as failed with the build’s error message. When Nix is unable to substitute a path, this is not an error, since that’s the normal prelude to building it, so a Substitute span with nix.build.status of NoSubstituters stays unmarked.

Distributed tracing

Nix rarely does all of its work in one process, so Determinate Nix propagates the W3C trace context wherever it hands work to someone else:

• The Nix daemon (when reached over ssh-ng://). The daemon opens a server span named daemon connection under the client’s trace, so everything the daemon does on your behalf appears in the same trace as the command that requested it.
• Remote builders. When Nix dispatches a build to a remote machine via build-remote, the nix-build-remote process and the remote daemon join the trace.
• Binary cache servers. Every HTTP request Nix makes carries a traceparent header, so a cache that participates in tracing can link its server-side spans to your trace.

Each process exports its own spans, so each process needs its own OTLP configuration. For a local daemon, the settings in /etc/nix/nix.custom.conf cover both the client and the daemon. For a remote daemon or builder, environment variables don’t survive the SSH hop, so configure the remote machine’s nix.custom.conf instead.

Debugging

Nix warns once per process if it can’t reach your collector, and an export failure never fails the command itself. Set NIX_DEBUG_OTEL=1 to have Nix print the root span’s trace ID to standard error when it exits, which makes it easy to find the trace in your backend. In that mode, Nix also sets sampling.priority = 1 on every span, which tells a collector’s probabilistic sampler to keep that trace.

NIX_DEBUG_OTEL=1 nix build .#my-package

OpenTelemetry trace ID: 0af7651916cd43dd8448eb211c80319c

Trying it locally

A quick way to see traces from Determinate Nix is to use otel-desktop-viewer, a single binary that accepts OTLP/HTTP and provides a simple trace viewer in the browser. You can run it with no configuration:

nix run "https://flakehub.com/f/NixOS/nixpkgs/0#otel-desktop-viewer"

In another terminal, fetch a package into an ephemeral directory:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
  nix build \
    --no-link \
    "https://flakehub.com/f/NixOS/nixpkgs/0#openjdk"

Open http://localhost:8000 and you should see a nix build trace from the nix service, with spans like BuildInstallables, CopySourcePath, and EvaluateFlakeDerivationOutput.

If you’re looking for a more detailed, programmatic view of traces, you can use the OpenTelemetry Collector instead. First, stop otel-desktop-viewer, then write a basic YAML configuration to a local file:

cat > otel-collector.yaml <<'EOF'
receivers:
  otlp:
    protocols:
      http:
        endpoint: localhost:4318
 
exporters:
  debug:
    verbosity: detailed
 
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [debug]
EOF

Then, in a new session, start the collector with that configuration set:

nix run "https://flakehub.com/f/NixOS/nixpkgs/0#opentelemetry-collector" -- \
  --config otel-collector.yaml

Then, in another session, fetch another package from Nixpkgs:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
  nix build \
    --no-link \
    "https://flakehub.com/f/NixOS/nixpkgs/0#cowsay"

To forward traces on to Jaeger, Grafana Tempo, or another backend, swap the debug exporter for an otlphttp exporter.