Determinate on Buildkite

Buildkite is a powerful CI/CD platform that is compatible with FlakeHub and FlakeHub Cache.

Buildkite onboarding

In order to use FlakeHub with Buildkite, you first need to acquire your organization’s Buildkite ID from the Buildkite API. First, generate a Buildkite API token with the read_organizations scope enabled, then query the API using that token. Here’s an example script that does that:

Query Buildkite API

# Add curl and jq to your current shell session
nix shell "https://flakehub.com/f/NixOS/nixpkgs/0.1#curl" "https://flakehub.com/f/NixOS/nixpkgs/0.1#jq"
 
# Set some required variables
buildkite_api_token="bkua_..." # Your generated API token
buildkite_org="..."            # Your Buildkite organization
 
# Query for the ID and display it
buildkite_organization_id=$(curl -H "Authorization: Bearer ${buildkite_api_token}" \
  "https://api.buildkite.com/v2/organizations/${buildkite_org}" | jq -r '.id')
 
echo "${buildkite_organization_id}"

At this stage, email support@determinate.systems with your organization ID and the name of your FlakeHub organization and we will notify you when your account has been enabled.

Example FlakeHub Cache pipeline

This Buildkite pipeline provides an annotated example of pushing store paths to FlakeHub Cache:

.buildkite/pipeline.yaml

steps:
  - label: Push store paths to FlakeHub Cache
    env:
      DETERMINATE_NIX_INSTALL_URL: https://install.determinate.systems/nix?ci=buildkite
      MAGIC_NIX_CACHE_CLOSURE_URL: https://install.determinate.systems/magic-nix-cache-closure/branch/main/X64-Linux?ci=buildkite
      MAGIC_NIX_CACHE_LISTEN: 127.0.0.1:37515
    command: |
      echo "Installing Determinate Nix"
 
      curl --proto '=https' --tlsv1.2 -sSf -L "${DETERMINATE_NIX_INSTALL_URL}" | \
        sh -s -- install linux \
          --determinate \
          --no-confirm \
          --init none
 
      nohup /usr/local/bin/determinate-nixd daemon &
      export NIX_REMOTE="daemon"
 
      echo "Waiting for Determinate Nixd status"
 
      while ! determinate-nixd status &>/dev/null; do
        sleep 0.1;
      done
 
      echo "Determinate Nix installed"
 
      # Set PATH to include Nix-created directories and persist that environment variable in a file
      PATH="${PATH}:/root/.nix-profile/bin/"
 
      echo "Fetching OIDC token for Buildkite"
 
      buildkite_oidc_token="$(\
        buildkite-agent oidc request-token \
        --claim 'organization_id' \
        --claim 'pipeline_id' \
        --audience "api.flakehub.com" \
      )"
 
      echo "Logging in to FlakeHub using OIDC token"
 
      echo "${buildkite_oidc_token}" | determinate-nixd login token --token-file /dev/stdin
 
      echo "Installing Magic Nix Cache"
 
      magic_nix_cache_closure="$(curl -L "${MAGIC_NIX_CACHE_CLOSURE_URL}" | xz -d | sudo "$(which nix-store)" --import | tail -n1 | head -n1)"
      sudo ln -sf "${magic_nix_cache_closure}/bin/magic-nix-cache" /usr/bin/magic-nix-cache
 
      magic_nix_cache_startup_file="/tmp/mnc-startup"
 
      echo "Waiting for Magic Nix Cache to start up"
 
      nohup magic-nix-cache --listen "${MAGIC_NIX_CACHE_LISTEN}" --startup-notification-file "${magic_nix_cache_startup_file}" &>/tmp/mnc.log &
      (
        started=0
        for n in {1..6}; do
          if [ -e "${magic_nix_cache_startup_file}" ]; then
            echo "Magic Nix Cache daemon has successfully started up after ${n} attempt(s)"
            started=1
            break
          else
            echo "Waiting on Magic Nix Cache daemon; on attempt ${n}"
            sleep 2
          fi
        done
 
        if [[ "${started}" != "1" ]]; then
          echo "The Magic Nix Cache daemon did not start up within 60 seconds; exiting"
          exit 1
        fi
      ) || true
 
      # Build whatever you'd like with Nix
      nix build .#my-package-1
      nix build .#my-package-2
 
      # Drain Magic Nix Cache
      curl -XPOST "http://${MAGIC_NIX_CACHE_LISTEN}/api/workflow-finish"
 
      echo "Store paths successfully pushed to FlakeHub Cache!"

Example FlakeHub publishing pipeline

This Buildkite pipeline provides an annotated example of publishing a Nix flake to FlakeHub as a tagged release:

.buildkite/pipeline.yaml

steps:
  - label: Published tagged release to FlakeHub
    env:
      DETERMINATE_NIX_INSTALL_URL: https://install.determinate.systems/nix?ci=buildkite
      FLAKEHUB_PUSH_BINARY_URL: https://install.determinate.systems/flakehub-push/branch/main/X64-Linux?ci=buildkite
      # Set these to match your own flake
      FLAKE_ORG: my-org
      FLAKE_NAME: my-flake
      FLAKE_VISIBILITY: unlisted # could also be public or private
    command: |
      echo "Installing Determinate Nix"
 
      curl --proto '=https' --tlsv1.2 -sSf -L "${DETERMINATE_NIX_INSTALL_URL}" | \
        sh -s -- install linux \
          --determinate \
          --no-confirm \
          --init none
 
      echo "Installing flakehub-push"
 
      curl -L "${FLAKEHUB_PUSH_BINARY_URL}" | sudo tee /usr/bin/flakehub-push &>/dev/null
      sudo chmod +x /usr/bin/flakehub-push
 
      echo "Fetching OIDC token for Buildkite"
 
      buildkite_oidc_token="$(\
        buildkite-agent oidc request-token \
        --claim 'organization_id' \
        --claim 'pipeline_id' \
        --audience "api.flakehub.com" \
      )"
 
      # Required environment variables for flakehub-push
      FLAKEHUB_PUSH_OIDC_TOKEN="${buildkite_oidc_token}"
      FLAKEHUB_PUSH_REPOSITORY="${FLAKE_ORG}/${FLAKE_NAME}"
 
      # Publish the flake if and only if it's a tag reference
      if [[ ! -z "${BUILDKITE_TAG:-""}" ]]; then
        flakehub-push \
          --tag "${BUILDKITE_TAG}" \
          --visibility "${FLAKE_VISIBILITY}" \
          --include-output-paths
 
        echo "The flake ${FLAKEHUB_PUSH_REPOSITORY} has been published to FlakeHub with version ${BUILDKITE_TAG}"
      fi

Determinate on Semaphore Migrating to FlakeHub Cache