Architecture

The problem

Some of our Godot tooling is compiled into the engine, not loaded as an addon. That means a project that needs module A but not module B needs its own engine binary, for every OS it ships on. Doing that by hand does not scale.

The pipeline

A build request is a BuildSpec: an engine ref, a platform/arch/target triple, and a list of modules. Everything downstream is a function of that value.

The modules a request may name are not open-ended. Each Godot major/minor has a stored list of the modules it offers and which version of each — see modules.md. Compatibility is a property of the engine version, held in the catalog, rather than a constraint each module carries and the builder re-evaluates.

  Svelte UI  ──POST /api/v1/builds──▶  gbb-server
                                          │
                                          ▼
                                    ┌───────────┐
                                    │   Queue   │  bounded worker pool
                                    └─────┬─────┘
                                          │
             ┌────────────────────────────┼────────────────────────────┐
             ▼                            ▼                            ▼
      ┌────────────┐              ┌──────────────┐            ┌───────────────┐
      │  Registry  │              │    Source    │            │    Runner     │
      │ list, pick │─────────────▶│  assemble    │───────────▶│  scons in a   │
      │ clone mods │              │  workspace   │            │  container    │
      └────────────┘              └──────────────┘            └───────┬───────┘
                                                                      │
                                                                      ▼
                                                              ┌───────────────┐
                                                              │ Artifact store│
                                                              └───────┬───────┘
                                                                      │
  Svelte UI  ◀──SSE logs, then GET .../artifacts/<name>────────────────┘

Stage	Package	Responsibility
Validate	internal/spec	Reject malformed specs before anything is cloned.
Resolve	internal/registry	Godot version + module names → that version's compatibility list → checkouts on disk.
Assemble	internal/source	Godot checkout with modules grafted into modules/.
Compile	internal/runner	Run SCons in the per-platform image over the Docker API, stream output, collect what lands in bin/.
Store	internal/artifact	Hash, store, and serve the produced binaries.
Track	internal/job, internal/store	Job lifecycle, and the Postgres catalog: modules, versions, compatibility lists, tokens.
Serve	internal/httpapi	REST API and the embedded frontend — api.md is the contract.

What a submitted build actually does

Since M3 the diagram above describes something that happens, so it is worth stating end to end.

POST /api/v1/builds decodes the spec, validates it, and checks the module selection against the chosen engine version's list — all before anything is queued, which is what makes an unoffered module a 400 in milliseconds rather than a failure an hour into a compile. The queue mints a ULID, writes a queued row, and hands the ID to a worker; the response is 202 and the job. The request is finished at that point, and the build is not: a worker's context comes from the pool, not from the HTTP request, so a build does not die because the client hung up.

A worker then, for one job: opens the log buffer and its file, moves the row to preparing, resolves the engine version's major/minor if modules were named, copies the assembler template and prepares the workspace, writes the resolved modules to the row before compiling anything, moves to building and runs SCons in a container, moves to packaging while the artifacts upload, and lands on succeeded. Any failure lands on failed with the reason; a cancel that was recorded before the context was signalled lands on canceled instead — the two are indistinguishable at the context, so the flag is what tells them apart.

Output goes two places at once: a bounded in-memory tail that the SSE endpoint replays to a client joining mid-build, and a file under <data_dir>/logs that is the complete record and outlives the process. A finished job's log streams from the file.

Why the catalog is a database, not a file

Modules, Godot versions, and the compatibility lists between them were YAML manifests loaded at startup. They are Postgres tables instead, for one reason: they are edited by people while the service runs. A privileged user adding a module, or pinning an exception on Godot 4.3's list, is an ordinary Tuesday — not a redeploy.

That also puts the compatibility lists somewhere they can be frozen. A list is generated once when its Godot major/minor is added and then only updated while that minor is the newest one, so what 4.3 offers records what was true when 4.3 was current. A file rewritten by a generator would have no way to hold that distinction, and a constraint evaluated per build would have nothing to hold it in.

Why containers

One Linux host produces all three targets:

• Linux — native GCC for x86_64, and Debian's crossbuild-essential-arm64 for arm64.
• Windows — mingw-w64, which Godot's build system supports directly. x86_64 only: Godot builds Windows on arm64 upstream with llvm-mingw, which Debian does not package, so the pair is refused at validation rather than failing in a container.
• macOS — osxcross, which needs an Apple SDK you supply, and MoltenVK, which is baked into the image because the build container has no network. See macos-sdk.md.

Each image is the authority on its own toolchain. internal/runner renders the build spec into an scons command line and nothing more; the image's entrypoint (build/docker/entrypoint.sh) adds what depends on that image — the cross compiler on PATH, the Apple SDK version osxcross was built against, whether a .app bundle is assembled. The orchestrator cannot inspect the image it is about to run, so a value hardcoded there would go stale the first time an image was rebuilt.

Builds are launched over the Docker Engine API using the official Go SDK (internal/runner.DockerRunner). The deployment target is Kubernetes and we ship the image, so the daemon endpoint is a known quantity rather than something to discover at runtime: it is a docker:dind sidecar in the same pod, reached at tcp://localhost:2375.

That arrangement has one consequence that shapes the code. A mount source is resolved by the daemon, not by the client. A path that exists only in the server container means nothing to the sidecar. So:

• assembled workspaces live on a volume mounted into both containers at the same path, and
• caches are named Docker volumes, whose location is the daemon's business.

deployment.md covers this in full; it is the single most likely thing to get wrong.

Why a Runner interface

runner.Runner has one implementation today. It exists for two futures that are already visible:

• Signing and notarization need codesign and notarytool, which only run on macOS. That means dispatching to a native runner, not a Linux container.
• Clusters that forbid privileged pods cannot host a dind sidecar at all, and would need builds dispatched elsewhere.

Either is a new implementation rather than a rewrite of the queue.

Reproducibility

BuildSpec.CacheKey() is a content hash of the normalized spec. Two requests with the same key must produce the same binary, which is what makes build deduplication (M7) safe. This is also why the registry records the commit SHA a ref resolved to, not just the ref — and why a Godot version's module list is stored and frozen rather than recomputed, so the same request a year apart draws from the same catalog.

It is not true yet. Two M1 builds of one unchanged spec, from the same commit in the same workspace path, produced binaries with different SHA-256 digests. The engine commit does reach the binary — a build reports 4.4.stable.custom_build.<hash> — but something else in the output varies. Until that is found, the key identifies a request rather than a byte-identical artifact, and nothing may rely on the stronger reading. Tracked in ROADMAP M7.

Deployment

Packaged as a container image (build/docker/server.Dockerfile) and deployed to Kubernetes as a StatefulSet with a privileged dind sidecar (deploy/k8s/). The server image carries git and openssh-client — it has to clone — but no build toolchain: it orchestrates, it does not compile.

Durable state is in Postgres and the object store, and since M6 the queue is a table rather than a channel in one process: a worker claims a job with SELECT … FOR UPDATE SKIP LOCKED, holds a lease it renews by heartbeat, and a cancellation is a column its owner notices rather than an in-memory signal that could only ever reach one process. A job whose lease lapses goes back on the queue, which replaced a startup pass that failed every non-terminal row — the single thing a second replica could least survive, since it would have killed the other replica's running builds.

That is one pin of three removed, and the service still runs one replica. ROADMAP.md M6 called the queue "the last thing pinning the service to one replica"; it was not, and this section said so before the work began. What is left:

• Live logs and the log archive are replica-local. Queue.Logs returns a buffer only the process running the build holds, and LogFile reads the local volume, so a replica that did not run a build cannot serve its log. Solving it means the archive in the object store, or an SSE proxy between pods — a milestone's worth of decisions, and not this one's.
• The build caches are on a ReadWriteOnce volume. The engine mirror, the module mirrors, and ccache are all local, and the mirror locks that keep two workers off one mirror are sync.Mutex values in one process.

deploy/k8s/statefulset.yaml stays at replicas: 1 for those two reasons. What the shared queue buys today is not scale-out but correctness: a restart no longer destroys in-flight work, and a cancel reaches whichever process owns the build.

Deliberate non-choices

• No cgo. The Postgres driver is github.com/jackc/pgx/v5, which is pure Go, so the server stays one static binary.
• No router dependency. Go's http.ServeMux handles method and path patterns since 1.22.
• Shell out to git. SSH keys, submodules, and LFS all work for free; pure-Go git libraries struggle with them. This has a cost the M0 image hid: distroless/static carries no git and no ssh, so a deployed server could never have cloned anything. The runtime stage moves to debian:bookworm-slim with git and openssh-client — the binary stays static, but distroless does not survive the choice. See docs/m2-module-registry.md.
• Frontend embedded in the binary. One image to deploy, and no separate static host to keep in sync with the API.