Deployment

The service is packaged as a container image and runs on Kubernetes. It orchestrates builds over the Docker Engine API and never compiles anything itself, so its own image carries no SCons and no cross-compilers — those live in the per-platform toolchain images it launches.

The shape

┌─────────────────────── pod ────────────────────────┐
│                                                    │
│  ┌──────────────┐         ┌────────────────────┐   │
│  │  gbb-server  │──API──▶ │  dind (privileged) │   │
│  │              │  :2375  │                    │   │
│  └──────┬───────┘         └─────────┬──────────┘   │
│         │                           │              │
│         │                           ▼              │
│         │                  ┌──────────────────┐    │
│         │                  │ build container  │    │
│         │                  │  gbb/linux, …    │    │
│         │                  │  scons           │    │
│         │                  └────────┬─────────┘    │
│         │                           │              │
│         ▼                           ▼              │
│      ┌────────────────────────────────────┐        │
│      │  data PVC, mounted at /data in     │        │
│      │  BOTH containers at the same path  │        │
│      └────────────────────────────────────┘        │
└────────────────────────────────────────────────────┘

Why a dind sidecar

Kubernetes has not used Docker as its container runtime since 1.24; nodes run containerd or CRI-O. There is no host Docker socket to mount, and even where one exists, handing a workload the node's socket is equivalent to giving it root on the node.

So the service ships its own daemon as a sidecar. It shares the pod's network namespace, which is why the endpoint is tcp://localhost:2375 with TLS off: nothing outside the pod can address that interface.

The sidecar needs privileged: true. There is no way around that for Docker-in-Docker. The consequences are worth stating plainly:

• Put this pod in its own namespace with its own service account.
• Schedule it on nodes you are willing to treat as a build farm, not alongside anything sensitive.
• If your cluster policy forbids privileged pods, this design does not fit, and the alternative is dispatching builds to native runners — which is what the runner.Runner interface exists to allow.

The bind-mount trap

A mount source is resolved by the daemon, not by the client.

When gbb-server asks the sidecar to run a build container with the workspace mounted, the path in that request is interpreted inside the dind container's filesystem. A path that exists only in the server container produces an empty directory, or a confusing failure, rather than an error naming the real problem.

Two consequences, both load-bearing:

1. Assembled workspaces live on a volume mounted into both containers at the same path. In deploy/k8s/statefulset.yaml the data PVC is mounted at /data in the server and in the sidecar. config.data_dir is /data, so WorkDir() means the same thing to both.

2. Caches are named Docker volumes, not host paths. build.ccache_volume and build.workspace_volume name volumes the daemon manages, so their location is the daemon's business and never has to line up with ours.

If a build ever produces an empty source tree, this is the first thing to check.

Storage

Finished artifacts go to the object store at store.picklemustard.dev, not to a volume — see artifacts.md. What remains on disk is only in-flight state.

Two PersistentVolumeClaims, both ReadWriteOnce:

Claim	Holds	Why it is large
data	Engine mirror, module mirrors, assembled workspaces, per-job logs	A Godot checkout is ~1 GB, and a workspace with build output several more
dind-storage	The sidecar's image and layer store	Toolchain images with a full cross-compiler run to several GB

The catalog and the job records live in Postgres, not on either claim — see below.

Job logs

Every build's complete output is written to <data_dir>/logs/<job-id>.log as it runs. The in-memory buffer that the log endpoint streams from holds only the last build.log_capacity lines (4096 by default) and dies with the process, so the file is what makes a finished job's log readable at all — and readable in full, not just its tail — after the build ends or the pod restarts.

These files are the one thing on the data claim that outlives the build that wrote it, and they are pruned on retention.logs — thirty days in the deployed ConfigMap — by the same collector that expires artifacts. Log pruning runs outside the collector's advisory lock, unlike everything else in that pass: the files are local to one replica's volume while the rows are shared, so under the lock only the lock holder would ever prune its own disk.

Rate limiting is per-replica, and the numbers in rate_limit are per-instance: with N replicas behind one Service the effective limit is N times what is configured. At replicas: 1 that is exact. It is held in memory on purpose — a shared limiter would mean a Redis or Postgres round trip on every request to bound something that is cheap to serve, which trades away the thing being protected. The durable, per-caller budget is elsewhere and is not per-replica: api_tokens.build_quota counts builds per window in the database, because an allowance belongs to a token rather than to a server.

A StatefulSet rather than a Deployment, because the disk state is local to one replica: two replicas on one RWO volume would fight over the same mirrors and workspaces. The queue is no longer among the reasons — it moved into Postgres in M6 — but the build caches and the per-replica log files still are. See architecture.md.

One consequence worth knowing for an ordinary restart: a build interrupted by one is no longer failed at startup. Its lease lapses after build.lease, the job returns to the queue, and it is picked up again — up to build.max_attempts times, after which it is failed with "interrupted by a server restart". Set max_attempts: 1 for the older behaviour, where such a build was failed and never retried. Moving artifacts to the object store and the database to Postgres removed most of what pinned this to one replica; what is left is the caches and the in-process queue. ROADMAP M6.

Postgres

The server needs a Postgres instance for the module catalog, the compatibility lists, the API tokens, and the job records. It is not optional and it is not embedded: gbb-server refuses to start without a reachable database, because every module and version endpoint is unanswerable without one and coming up degraded only moves the failure somewhere less visible.

Supply the DSN as GBB_DATABASE_DSN from the gbb-postgres Secret:

postgres://gbb:<password>@postgres.gbb.svc.cluster.local:5432/gbb?sslmode=require

Migrations are embedded in the binary and applied at startup, so a fresh database needs no separate step. The driver is pgx, which is pure Go — the database costs the image nothing.

Nothing here provisions Postgres itself. Point it at whatever the cluster already runs; the schema is small and the write rate is a handful of rows per build.

Secrets

Three Secrets, none of them in the ConfigMap. deploy/k8s/secret.example.yaml ships placeholder values for all of them and must be replaced before anything is applied.

Secret	Holds	Reaches the server as
gbb-s3	Object store credentials	GBB_S3_ACCESS_KEY_ID, GBB_S3_SECRET_ACCESS_KEY
gbb-postgres	Database DSN, admin bootstrap token	GBB_DATABASE_DSN, GBB_ADMIN_BOOTSTRAP_TOKEN

Each of those credentials has two other routes into the process, for a deployment that would rather not put a secret in an environment variable or would rather review the non-secret half of a connection in git:

• Describe the connection in the ConfigMap, keep only the password in the Secret. Set database.host, port, user, name and sslmode in config.yaml and project just GBB_DATABASE_PASSWORD. The password is percent-escaped when the connection string is composed, which a hand-written DSN is not — a password containing @ or / makes one parse as a different host. Set one form or the other; a config with both a dsn and a host is refused rather than silently resolved.
• Mount the Secret and name the path. database.dsn_file, database.password_file, redis.url_file, redis.password_file and artifacts.s3.secret_access_key_file read the value from a file, which keeps it out of the environment entirely. Trailing newlines are trimmed. A path that cannot be read is fatal at startup. | gbb-modules-ssh | Deploy key and known_hosts for private module repos | A read-only volume at /etc/gbb/ssh |

The module deploy key

gbb-modules-ssh holds id_ed25519 and known_hosts, mounted read-only at /etc/gbb/ssh with defaultMode: 0400. This is not tidiness: ssh refuses a group- or world-readable private key outright, and Kubernetes' default 0644 will fail every private clone with an error that does not obviously point at file permissions.

modules:
  ssh_key_path: "/etc/gbb/ssh/id_ed25519"
  known_hosts_path: "/etc/gbb/ssh/known_hosts"

The build container's uid

build.container_user is not optional under Kubernetes, and the failure it prevents names nothing that points at it.

The toolchain images end in USER builder (uid 1000) so that a local build's output belongs to the caller. In this deployment the server runs as uid 65532 and creates /data/workspaces/<job> at mode 0755, then hands that directory to the build container as a bind mount. uid 1000 cannot write into it, so SCons dies on the first object file with a permission error about a path in the engine tree.

build:
  container_user: "65532:65532"

"0:0" also writes, and is what rootless podman needs for the opposite reason (see below) — but here it leaves root-owned directories on the PVC that Cleanup's RemoveAll, running as 65532, cannot remove. A Godot workspace is several gigabytes, and it would leak one per build. Match the uid instead.

The ccache named volume has the same problem from the other end: Docker initialises a new volume from the image's /ccache, so it is owned by whichever uid the image used — or by root, if the image has no such directory. A volume the build container cannot write is a cache that silently never caches.

known_hosts is required whenever a key is set. Without it, StrictHostKeyChecking either fails opaquely or waits for a prompt no worker can answer — which on a build node is indistinguishable from a hang.

The bootstrap token

GBB_ADMIN_BOOTSTRAP_TOKEN seeds the first admin token when api_tokens is empty, and only then. It exists so a fresh database has a way in. Issue a real token and revoke it — admin.md has the two calls. Every use of it is logged loudly.

The server image

Every merge to main builds build/docker/server.Dockerfile and pushes it to the Forgejo package registry as git.picklemustard.dev/studioteal/gbb-server, tagged :latest and with the twelve-character commit SHA. The push only runs if the Go and frontend jobs on that same merge commit passed — see .forgejo/workflows/ci.yml (.forgejo/workflows/ci.yml in the repo).

The job authenticates with the FORGEJO_API_TOKEN secret, which needs write:package on the owning organisation alongside its repository scopes; the login username is read out of the token by scripts/forgejo-actor.sh rather than hardcoded. Building the image by hand is still make server-image, which tags it gbb/server: locally and pushes nothing.

The toolchain images below are not built on every merge — they are multi-gigabyte and change rarely, so that would be waste. A release does publish them, at its own vX.Y.Z, and skips the rebuild when build/docker/ is unchanged since the previous release.

Loading the toolchain images

The build images must be in the sidecar's image store, not the node's. Either:

• let the daemon pull them on first use — the normal answer, and what a release now makes possible; or
• docker save them and load them into the sidecar (deploy/local/load-toolchain-images.sh), for an air-gapped cluster or a Dockerfile you have not pushed.

For the first, three settings have to agree: build.image_prefix naming the registry, build.image_tag naming the version, and a build.registry credential. That last one is easy to skip and hard to diagnose — the pod's imagePullSecret does not cover it. See deploy/k8s/README.md (deploy/k8s/README.md in the repo) and the note in deploy/k8s/configmap.yaml.

Building them by hand is scripts/build-images.sh, on a build machine rather than inside the cluster. DOCKER selects the client and IMAGE_PREFIX the tag prefix, so podman works:

DOCKER=podman IMAGE_PREFIX=localhost/gbb/ ./scripts/build-images.sh

IMAGE_TAG sets the tag (default latest) and PUSH=1 pushes what it builds; scripts/publish-toolchain-images.sh is the release wrapper around exactly that, and is also how a machine holding the Apple SDK publishes gbb-macos, which CI cannot build.

Whatever prefix is used has to match build.image_prefix, and the tag build.image_tag; that is what runner.ImageTag assembles when it names the image to run. make images passes both variables through, which it did not before M5 — make does not export its variables into a recipe's environment, so make images IMAGE_PREFIX=… used to build under the script's own default and produce images the config could not find.

ImageTag is keyed on platform alone, and stays that way: each image carries every architecture it can build, so there is no gbb/linux-arm64 to load separately. That matters most here, where every extra multi-gigabyte image is something a sidecar has to be given.

Configuration

deploy/k8s/configmap.yaml supplies config.yaml, mounted read-only at /etc/gbb. It no longer carries module or Godot-version manifests: both are Postgres tables, edited over the admin API rather than by redeploying a ConfigMap. Anything can also be overridden by a GBB_-prefixed environment variable, which is the easier route for per-environment differences:

Variable	Effect
GBB_ADDR	Listen address
GBB_DATA_DIR	Must match the mount path in both containers
GBB_WORKERS	Concurrent builds
GBB_JOBS	SCons -j per build
GBB_QUEUE_CAPACITY	Queued builds accepted before POST /builds answers 503
GBB_LOG_CAPACITY	Lines of a running build's log kept in memory for replay
GBB_DOCKER_HOST	Daemon endpoint (or the standard DOCKER_HOST)
GBB_WORKSPACE_VOLUME	Docker volume holding build trees
GBB_WORKSPACE_MOUNT	volume or bind; see Local development
GBB_CONTAINER_USER	Override the build container's user, e.g. 0:0
GBB_CCACHE_VOLUME	Docker volume for ccache
GBB_S3_ENDPOINT, GBB_S3_BUCKET, GBB_S3_PREFIX	Object store location
GBB_S3_ACCESS_KEY_ID, GBB_S3_SECRET_ACCESS_KEY	Object store credentials
GBB_S3_SESSION_TOKEN, GBB_S3_PROFILE	Temporary credentials, or a shared-credentials identity
GBB_S3_SECRET_ACCESS_KEY_FILE	Read the secret key from a mounted file instead
GBB_S3_CA_BUNDLE_PATH	Trust a private CA for a self-hosted store
GBB_ARTIFACT_BACKEND	s3 or fs
GBB_DATABASE_DSN	Postgres connection string. Required, unless the parts below are set
GBB_DATABASE_HOST, _PORT, _USER, _NAME, _SSLMODE	The connection described in parts, instead of a DSN
GBB_DATABASE_PASSWORD, GBB_DATABASE_PASSWORD_FILE	The password on its own
GBB_DATABASE_MAX_CONNS, GBB_DATABASE_CONNECT_TIMEOUT	Pool tuning, previously ConfigMap-only
GBB_HUMAN_REDIS_URL	Verification and enforcement Redis; public mode only
GBB_REDIS_HOST, _PORT, _USER, _PASSWORD_FILE, _DB	The same, described in parts
GBB_ADMIN_BOOTSTRAP_TOKEN	Seeds the first admin token on an empty database
GBB_MODULES_SSH_KEY_PATH, GBB_MODULES_KNOWN_HOSTS_PATH	Module deploy key

Resources

A Godot build is memory-hungry at link time and will use every core it is offered. The manifests cap the build container at 8 GiB and 4 CPUs (build.memory, build.cpus), and the sidecar's own limit must exceed that — the build container's usage counts against the sidecar's cgroup, not the server's.

Keep build.workers at 1 until you have measured a node running two concurrent builds without the OOM killer intervening.

Local development

No Kubernetes needed, and no dind: locally the daemon runs on this machine and shares our filesystem, which removes the constraint the whole arrangement above exists to work around. build.workspace_mount: bind says so, and the workspace directory is then mounted directly.

make dev runs the server against whatever DOCKER_HOST points at. Podman's Docker-compatible API works unchanged:

systemctl --user enable --now podman.socket
export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock

Build the toolchain images, then run a build without the server at all:

DOCKER=podman IMAGE_PREFIX=localhost/gbb/ ./scripts/build-images.sh
./bin/gbb build --godot 4.4-stable --platform linuxbsd --target editor --local

--local runs the whole pipeline in the CLI — assemble, compile, collect — with no queue and no database, and writes the binaries under ./out. It is the shortest path between a change to the pipeline and knowing whether it works.

Three things about podman are not interchangeable with Docker, and all three show up as failures that name something other than their cause:

Setting	Podman value	Why
build.image_prefix	localhost/gbb/	Podman stores locally built images under localhost/ and resolves anything unqualified against docker.io. gbb/linux:latest is a registry lookup that fails, not the image just built.
build.image_tag	latest	What scripts/build-images.sh tags with unless IMAGE_TAG says otherwise. Pin it to a vX.Y.Z only if you are pulling a published set.
build.container_user	"0:0"	Rootless podman maps the invoking user to container root and container UID 1000 to a subuid. The images run as UID 1000 so output belongs to the caller; rootless podman inverts that, and 0:0 is what restores it.
SELinux	handled in code	On Fedora and RHEL a bind mount without a :z relabel is not writable. DockerRunner.hostConfig always requests it; the flag is ignored where SELinux is not enforcing.

The engine mirror is bind-mounted read-only alongside the workspace, because a git worktree's .git is a file pointing into the mirror and Godot's SConstruct follows it to stamp the engine commit into the binary. With the mirror mounted a finished build reports 4.4.stable.custom_build.4c311cbee; without it, a bare 4.4.stable.custom_build and a "not a git repository" line in the log. Under volume mode the mirror is out of reach and that stamp is empty either way; Workspace.GodotCommit records the commit regardless.

A daemon it cannot reach is fatal at startup, as of M3. The server's job is to run builds; coming up without a daemon only moves the failure somewhere less visible than the startup log. The same now goes for the artifact bucket and, as since M2, the catalog database — every external dependency is checked before the listener opens.

A local cluster

make dev above removes Kubernetes from the picture, which is the right trade for working on the pipeline and the wrong one for finding out whether the deployment works — the dind sidecar, the bind-mount path agreement, the uid, the probes and the ingress are exactly what it skips.

deploy/local (deploy/local in the repo) is a kustomize overlay over deploy/k8s for a single-node k3s or k3d cluster. It pins the released image tags, points Postgres, Redis and the object store at ones you already run (host.k3d.internal from inside k3d, never localhost), clears modules.ssh_key_path so no deploy-key Secret is needed, sets auth.cookie_secure: false because the cluster is reached over plain HTTP, sizes the build container and the PVCs for a laptop, and retargets the gateway's NetworkPolicy at kube-system, where k3s actually runs Traefik.

It does not provision Postgres, Redis or MinIO, and it leaves anonymous mode off: GBB_HUMAN_ENABLED requires an HTTPS GBB_PUBLIC_ORIGIN and Turnstile keys, neither of which exists locally. The one step with no substitute is loading the toolchain images into the sidecar's own image store — deploy/local/load-toolchain-images.sh does it. Full runbook in deploy/local/README.md (deploy/local/README.md in the repo).

Operating a server that runs builds

Logs. Every build's complete output goes to <data_dir>/logs/<job-id>.log as it runs; see Job logs above. Nothing prunes them.

build.workers. Defaults to 1, and should stay there until someone measures. Each worker saturates build.jobs cores and wants several GB while linking, so two workers on one node is two full Godot builds competing for the same CPU, the same ccache volume, and the same SCons cache. The engine mirror is safe — the assembler serialises fetches per mirror, the way the registry serialises module fetches — but the shared caches are not, and that is ROADMAP M7's "concurrent builds do not corrupt the shared caches". The single ReadWriteOnce volume and single replica are the other reason to leave it alone.

build.queue_capacity. How many jobs may wait for a worker. A submission beyond it is refused with 503 rather than queued, deliberately: a persisted job in queued that no worker will ever take is the one outcome worth failing loudly to avoid.

A restart during a build. The container dies with the pod, so the row it left behind is a lie. On startup the queue fails every job it finds in preparing, building, or packaging with interrupted by a server restart, and pushes every queued job back onto the channel, oldest first. A crash-looping pod therefore does not rebuild the same job forever. SIGTERM is gentler: the listener stops, in-flight builds are canceled, their containers are removed by the runner's own teardown, and their terminal rows are written before the pool closes — with a 60-second cap, because the point is an orderly failed rather than waiting out a three-hour compile.

A completed job's log survives all of this, because it is on disk rather than in the buffer. A build that finished before the last restart still streams in full.

Status

Linux and Windows builds work end to end, but only through gbb build --local and only against a daemon on this machine. Verified: the toolchain images build, a stock Godot editor binary comes out of a container and into the artifact store with its checksum, output streams while the build runs, and Ctrl-C removes the container and the workspace rather than orphaning them.

Windows joined that list in M5 — a 4.4-stable editor cross-compiles through mingw-w64 in about ten minutes and yields both the gui and console executables. Linux arm64 cross-compiles through Debian's toolchain. The macOS image has still never been built, because osxcross needs an Apple SDK that cannot live on a shared machine (docs/macos-sdk.md).

The server image builds and runs: it serves the API and the embedded frontend as a non-root user, and refuses to start when the database, the Docker daemon, or the artifact bucket is unreachable.

Since M3 it runs builds. POST /api/v1/builds queues a job, a bounded worker pool drives it through preparing → building → packaging → succeeded, the log streams over SSE while it runs, and the binary lands in the object store. See api.md for the contract and "Operating a server that runs builds" below for what that costs to run.

The catalog and Postgres landed in M2. internal/store opens a pool, applies its embedded migrations at startup, and serves the module, Godot-version, and admin endpoints; the YAML manifests this document used to describe are deleted. The runtime image is debian:bookworm-slim with git and openssh-client, so it can clone — the previous distroless/static image carried neither and could not have assembled a workspace at all.

The first cluster run (M3-13)

The manifests have now been applied to a real three-node k3s cluster (v1.34, containerd, Traefik). What follows is what happened, in the M1 deviations format: the run found four things, three of which are fixed above.

The service comes up, and every dependency check works. The server connects to Postgres, reaches the artifact bucket at store.picklemustard.dev, and pings the dind daemon, and it refuses to start when any of the three is missing — each of those failures was observed in turn on the way to a working deployment, which is a better test of "fatal at startup" than the happy path. Both probes go green and the pod settles at 2/2 Running with no restarts.

The dind sidecar bound port 2375 twice. With DOCKER_TLS_CERTDIR empty the docker:28-dind entrypoint adds its own --host=tcp://0.0.0.0:2375, so the manifest's --host=tcp://127.0.0.1:2375 became a second listener on the same port and the daemon died with failed to allocate daemon listening port 2375 (err: Bind for 0.0.0.0:2375 failed: port is already allocated). Dropping the manifest's flag would have "fixed" it by leaving an unauthenticated root Docker socket on the pod IP, reachable from anywhere in the cluster. The sidecar now runs dockerd directly, bypassing the entrypoint, and keeps its loopback bind.

Nothing sequenced the server behind the daemon. The server treats an unreachable daemon as fatal (M3-10), the two containers started together, and the server lost the race every time — the pod crash-looped its way to readiness through the restart backoff. The sidecar is now a native sidecar: an init container with restartPolicy: Always and a startup probe, so the daemon is answering before the server's first connection attempt. This needs Kubernetes 1.29 or newer.

The server container was OOMKilled mid-clone. The engine mirror is cloned in the server container, because build containers have no network — and a bare clone of Godot does not fit in the 512Mi limit the manifest shipped. The container died with exit 137 about eighty seconds into the first build, and the job it was preparing came back as failed with interrupted by a server restart. The limit is now 4Gi, which is headroom for git rather than for the server. The restart reconciliation being proven by accident is the one good thing about that failure.

The mirror is bigger than the code claimed. du -sh on the finished mirror reports 6.2 GB, against the "~2 GB" the clone message and two comments carried. Those now say ~6 GB. It also means the data volume's shipped 50Gi is right and not generous: the mirror alone is an eighth of it before any workspace exists.

SSE survives the ingress. Streamed through Traefik from outside the cluster, the response keeps Content-Type: text/event-stream, X-Accel-Buffering: no, and Transfer-Encoding: chunked, and lines arrive as they are produced rather than in one block at the end. Both the live path and the finished-job file replay were checked this way.

Per-job log files survive. /data/logs holds one <id>.log per job, including the job whose server was OOMKilled out from under it.

workspace_mount: volume does not work, and cannot as written. This is the thing the run was for. Volume mode asks the daemon to mount subpath <job-id> of build.workspace_volume, but the assembler writes the workspace to <data_dir>/workspaces/<job-id> on our own filesystem and nothing ever copies it into that Docker volume. The daemon looks inside an empty volume and the build dies:

start build container: Error response from daemon: cannot access path
/var/lib/docker/volumes/gbb-workspaces/_data/<job-id>: no such file or
directory

It is not a configuration mistake; there is no configuration that fixes it. Volume mode would need a transfer step — copying the assembled tree in through the daemon — that does not exist. The ConfigMap now sets workspace_mount: bind, which is what this pod was always built for: the data PVC is mounted at /data in both containers precisely so a workspace path means the same thing to each. Two tests asserted the broken arrangement and now assert this one.

The artifact round trip works, after a fix. With bind mode a build runs to succeeded and the artifact reaches the bucket — but its recorded SHA-256 came back empty. S3Store.Put uploaded and then Stated the object to learn the digest, and a multipart upload's ChecksumSHA256 is a checksum-of-checksums, not the file's digest, so there was nothing to record. Every artifact large enough to go multipart — which is every artifact — was stored with no digest, and the download-time comparison had nothing to compare against. Put now computes the digest with an io.TeeReader as the bytes stream past, which is what its own doc comment had claimed all along.

Verified end to end afterwards, against the real endpoint:

GET /api/v1/builds/{id}/artifacts/{name}
  -> 302 https://store.picklemustard.dev/godot-builds/artifacts/...
  -> 41943040 bytes, etag "7425a2c44bb8ec33a15b0a11eb3a5867-8"
  recorded   e989ec92d979da6b443817dcf74a4daa8286e1ec16c41c920b24391fe0aad0fc
  downloaded e989ec92d979da6b443817dcf74a4daa8286e1ec16c41c920b24391fe0aad0fc

The -8 on the ETag is the point: eight parts, a genuinely multipart object, and the digest still matches. internal/artifact's gated live test covers this now and has been run against store.picklemustard.dev; it previously asserted that an empty digest was correct, so running the gate would not have caught it.

The compile is still a fake. The build above ran a stand-in toolchain image that writes 40 MiB where SCons would, not a Godot compile. What that proves is the whole pipeline around the compiler — prepare, the bind mount, the container lifecycle, collect, upload, the presigned redirect, the digest. What it does not prove is that Godot builds in-cluster, or anything about build time, memory, or ccache. A real toolchain image and a real compile remain outstanding, and with them M3-15 (tickets/m3-15-deferred-m2-acceptance.md in the repo), cancel-mid-compile, and the engine-commit stamp question. See M3-13 (tickets/m3-13-cluster-verification.md in the repo).

The artifact store has not been exercised against the real endpoint at store.picklemustard.dev. The server has produced artifacts since M3-10; what is missing is credentials and a cluster to run a build in. The filesystem backend and the download handler are covered by tests, and the digest the handler compares is recorded during collection precisely because S3 reports a checksum-of-checksums for a multipart upload — see artifacts.md. That comparison is the part the round trip has to prove.

The manifests render under kubectl kustomize but have not been applied to a live cluster, and no build has ever run through a dind sidecar. The workspace_mount: volume path — the named volume with a per-build subpath, the whole reason for the arrangement described above — is therefore still reasoned from how the Docker API resolves mount sources rather than demonstrated. Only bind has been exercised — by the daemon-backed tests in internal/runner and internal/job, which run against a local socket. Tracked as M3-13 (tickets/m3-13-cluster-verification.md in the repo).