Roadmap

Milestones are sequential. Each one has acceptance criteria that must hold before the next begins.

---

M0 — Scaffolding ✅

Repository, documentation, directory structure, and skeleton code that compiles and runs.

Acceptance

• go vet ./... and go build ./... are clean
• npm run build produces web/build/
• gbb-server listens and answers GET /healthz with {"status":"ok"}
• gbb-server serves the embedded frontend at /
• Unimplemented endpoints return 501, not a panic
• gbb build --dry-run validates a spec and prints it
• README, architecture, module, and macOS SDK docs written
• Git initialised on main with an initial commit
• Docker Engine API client connects at startup and logs the daemon version
• build/docker/server.Dockerfile produces a distroless image that serves the API and the embedded frontend as a non-root user
• Kubernetes manifests render under kubectl kustomize (not yet applied)
• Artifact storage implemented for S3 and the filesystem, with the download handler redirecting to a presigned URL and falling back to streaming

---

M1 — One real Linux build ✅

Compile stock Godot for Linux in a container. No modules yet.

Work

• source.Assembler.Prepare — mirror-clone the engine into the cache, create a worktree for the requested ref
• DockerRunner.Build — create the container, stream ContainerLogs (stdout and stderr demultiplexed), wait on ContainerWait, remove it afterwards. Cancellation must remove the container, so teardown needs its own context.
• Pull the toolchain image when absent, streaming pull progress into the log
• Collect the binaries out of the finished container and into the artifact store, recording the SHA-256 computed during collection (S3 multipart uploads cannot supply it; see docs/artifacts.md)
• Build and verify gbb/base and gbb/linux
• build.workspace_mount — a bind mode for a daemon that shares our filesystem, alongside the volume-and-subpath arrangement dind needs
• gbb build --local to drive the pipeline without a server

Acceptance

• gbb build --godot 4.4-stable --platform linuxbsd --local produces a Godot editor binary that launches
• Build output streams to the terminal as it happens, not at the end
• Ctrl-C kills the container, not just the client
• A second build of the same spec reuses the engine mirror
• The same build succeeds against a dind daemon, not only a local socket — this is what proves the mount arrangement, and a local-socket-only pass does not. Deferred to M3, which is where a cluster first exists to run it against — and still not run: M3 built everything needed to run it and stopped at the cluster. workspace_mount: volume is written and unit-tested and has never executed. See docs/tickets/m3-13-cluster-verification.md.

Deviations from the plan above

• Artifacts land in ./out/<job-id>/, not flat in ./out, because --local reuses artifact.FSStore rather than copying files itself — which is also where the SHA-256 comes from.
• The engine mirror is bind-mounted read-only next to the workspace. A worktree's .git is a file pointing into the mirror, and Godot's SConstruct follows it to stamp the engine commit into the binary. Under volume mode the mirror is out of reach and that stamp is empty.
• Three podman-specific settings turned out to be load-bearing rather than cosmetic: the localhost/ image prefix, container_user: "0:0" for the inverted rootless UID map, and an SELinux :z relabel on the bind mount. docs/deployment.md has the table.

---

M2 — Module catalog and registry

Resolve, cache, and graft modules — and first, put the catalog somewhere a person can edit it.

Compatibility is a property of the Godot version, not of a module: each Godot major/minor has a stored list of the modules it offers and which version of each. The list is generated once when that version is added and updated only while it is the newest one, so what 4.3 offers records what was true when 4.3 was current. docs/m2-module-registry.md is the design; docs/modules.md and docs/admin.md are the two user-facing halves.

Work — catalog

• Postgres via pgx, replacing the unimplemented SQLite store. Embedded migrations applied at Open; the database is fatal at startup
• Schema: modules, module_versions, module_version_requires, godot_versions, godot_module_compat, api_tokens, jobs
• Generation and freezing: adding a Godot major/minor generates its list; adding a module version refreshes only the newest major/minor; origin='admin' rows survive regeneration
• Delete internal/manifest, configs/modules.yaml, and configs/godot-versions.yaml
• Admin API for modules, module versions, Godot versions, and compat rows, behind bearer tokens with an is_admin flag — M6's model built early and scoped to the admin surface
• GET /api/v1/modules?godot_ref=<ref> returns that version's list; the bare form is a 400

Work — registry

• internal/gitcmd — one git invoker shared by the assembler and the registry, carrying the deploy-key environment
• registry.Select — draw from one minor's list, expand requires transitively, distinguish an unknown module from an unoffered one
• registry.Resolve — mirror or fetch each module, record the resolved commit
• registry.Export — git archive the tree into the workspace at its DestPath, .git excluded, with tar-slip and non-empty-dest guards
• Deploy keys: gbb-modules-ssh at mode 0400, and a runtime image that actually has git and ssh — distroless/static has neither

Acceptance

• A build with two modules, one requiring the other, compiles both in. The pipeline is verified end to end against the fixtures -- selected, resolved, exported, grafted at modules/<name> -- but the SCons compile itself has not been run. M3 removed the blocker (there is a server that runs builds now) without running it; the measurement is docs/tickets/m3-15-deferred-m2-acceptance.md
• Godot 4.4's list offers a module at v0.4.0 while 4.3's still says v0.3.1, after v0.4.0 is added. This is the design's whole point and the one thing to check by hand
• A fork ref (studioteal/4.4, recorded as major 4 minor 4) offers exactly what 4.4-stable offers, rather than silently skipping the check as the constraint design did
• Selecting a module the chosen Godot version does not offer fails immediately, naming the module and its minimum, before any cloning
• An unknown module name fails before any cloning happens
• Every admin route rejects a missing, non-admin, and revoked token identically
• --only-selected produces a measurably smaller binary. No new code -- the plumbing is in internal/runner/scons.go -- but the two builds have not been run. Same blocker, same ticket: M3-15
• The resolved commit SHA and module version of every module is recorded on the job

Deviations from the plan above

• The freeze needed a rule the design did not state. "Adding a module version refreshes only the newest major/minor" is what the design says, and implementing it literally made its own acceptance criterion unreachable: a Godot minor registered before a module exists generates an empty list, and under that rule nothing could ever join it — 4.3 offered nothing, forever. The rule that actually holds is: a list with no row for a module gets one generated; a list that already has a row is frozen, and only the newest minor moves. The freeze protects a recorded decision, and a module absent from a list never had one. Section 11 step 4 passes exactly as written.

• Rule 2 orders by recency, not by minimum. "The newest version whose minimum is at or below the engine" was first read as "the one with the highest minimum". Those diverge the moment a release lowers its minimum: v2 at min 4.3 added after v1 at min 4.4 is still the newer release, and it is what 4.4 should offer.

• internal/catalog exists because the ticketed import direction is impossible. M2-08 asks for CatalogEntry in registry, returned by store, while also requiring that store not import registry — the two cannot both hold. And registry -> store closes a cycle through job -> runner -> source. The shared value type and the two selection errors live in a leaf package both sides import.

• A DSN is not required to load a config. M2-03 asked for it in Validate, which would have broken M2-17's guarantee that gbb build --local works with no Postgres. The requirement moved to Config.RequireDatabase(), asked by gbb-server at startup and by the CLI only when modules are requested. The database is still fatal for the server.

• The KnownFields(true) warning in the tickets was about the wrong loader. config.Load uses a plain yaml.Unmarshal and ignores unknown keys; only the deleted manifest package was strict. A stale key in config.yaml was never a hard error, and the deployed ConfigMap is now guarded by a test that loads it through config.Load instead.

• Test fixtures ignore the developer's git config. A global commit.gpgsign = true makes every fixture commit invoke gpg, which blocks on a pinentry prompt no test can answer — the suite hangs rather than fails. modfixture.GitEnv() sets GIT_CONFIG_GLOBAL=/dev/null, which also fixed a latent hang in the M1 internal/source tests and cut internal/gitcmd from 4.4s to 0.02s.

• The tar-slip guard was wrong on the first attempt, and the hand-built-tar test M2-12 insisted on is what caught it: cleaning ../../escape to escape keeps the write inside the destination but creates a file the archive never described. Traversal is now refused rather than sanitised.

• The admin surface is fourteen routes, not the thirteen M2-15's prose claims; its own list enumerates fourteen.

• job.ModuleResolution landed with M2-05 rather than M2-18, because 0002_jobs.sql carries the resolved_modules column and the store could not compile without the type. The converter is in internal/job, not internal/registry: the reverse edge would close a cycle.

---

M3 — Server core

The service becomes usable over HTTP. The code has landed; the two items that need a live cluster and a real multi-hour compile have not been run. They are the unticked boxes below, and nothing else in the milestone is blocked on them.

Work

• job.Store over the Postgres pool M2 opened — landed in M2-05 with the jobs migration, and already wired; M3 only confirmed it
• Queue.Start / Submit / Cancel — bounded worker pool driving each job through preparing → building → packaging
• Every remaining handler in internal/httpapi/builds.go
• Make an unreachable Docker daemon fatal at startup rather than a warning
• Make an unreachable artifact bucket fatal at startup rather than a warning
• Verify uploads and presigned downloads against the real endpoint at store.picklemustard.dev — not run; needs credentials and a build that produces something to upload. See docs/tickets/m3-13-cluster-verification.md
• SSE log streaming, replaying the retained tail before live lines
• Per-job log files so a completed job's full log survives a restart
• LogBuffer.Write splitting on newlines rather than per write
• (Beyond the original list) the CLI's five unimplemented commands, and internal/client under them
• (Beyond the original list) docs/api.md — the contract M4 is built against

Acceptance

• POST /api/v1/builds returns 202 and a job ID
• GET /api/v1/builds/{id}/logs streams a running build's output live
• Cancel stops a running build and the job lands in canceled
• Jobs survive a server restart with their status intact — in-flight jobs are failed with interrupted by a server restart, queued ones are requeued
• The module resolutions M2 records reach the persisted job, written before the compile starts so a build that fails an hour later still carries them
• GET .../artifacts/{name} redirects to the object store and the downloaded binary's SHA-256 matches the record — run against store.picklemustard.dev in the M3-13 cluster run, on a genuinely multipart object (40 MiB, ETag suffix -8). It did not match at first: S3Store.Put recorded an empty digest for every multipart upload, which is every artifact. Fixed, and the gated live test that asserted the empty digest was correct now asserts the real one
• No endpoint returns 501 any more; notImplemented is deleted
• The service comes up under kubectl apply -k deploy/k8s and completes a build in-cluster — half run. It comes up: applied to a three-node k3s cluster it reaches 2/2 Running, both probes green, with Postgres, the real artifact bucket, and the dind daemon all connected. It also completes a build against a stand-in toolchain image: prepare, the bind mount, the container lifecycle, collect, the upload, and the artifact round trip all work. Getting there took five fixes, recorded in docs/deployment.md. What is missing is a real Godot compile — no toolchain image exists in a registry the cluster can reach — so build time, memory, ccache, cancel-mid-compile, and the engine-commit stamp question are all still unanswered

Deviations from the plan above

• Cancel takes the per-job lock before it looks at the running set, and never writes a running job's row. store.Postgres.Update is a whole-row write with no compare-and-set, so two writers silently clobber each other rather than conflicting. The rule the implementation settled on is stated in Queue's doc comment: the queue is the sole writer of a job row, and a running job is written only by its own worker. Cancel signals and returns; the worker records the terminal state.

• The cancel-during-packaging race has a fallback the tickets did not anticipate. M3-07 says a cancel arriving in packaging is a 409, which it is — but the check and the worker's own transition are not atomic, and a cancel accepted a moment before packaging begins would leave the worker trying an illegal packaging → canceled. The worker falls back to failed in that window rather than the state machine being widened.

• Status writes use a context detached from the caller's. Not in any ticket, and the whole shutdown path depends on it: Start's context is canceled to stop the builds, and every terminal status write happens after that. context.WithoutCancel plus a 30-second bound is what makes "a SIGTERM mid-build leaves a terminal row" true rather than aspirational.

• NewQueue takes a config struct, not a parameter list. M3-07 asked for a constructor "rather than exported fields so nothing can be half-built". With eleven dependencies a positional constructor is unreadable, so it is NewQueue(QueueConfig) (*Queue, error) — still one entry point, still validating, and it returns an error instead of panicking on a missing store.

• The queue does not hold an artifact.Store. The old Queue had one and never used it: the runner uploads each artifact as it collects it and returns the refs. The field is gone rather than carried.

• source.NewAssembler exists because lazy lock initialisation is not enough. M3-04 asked for a lock map behind a pointer field so per-job copies share it. Initialising it lazily on first Prepare is racy in exactly the case the locks are for — two goroutines on one assembler each install their own map, and the second erases the first. There is now a constructor that installs it up front, and a package-level mutex guarding the lazy path for an assembler built as a struct literal.

• docs/api.md was written after the handlers, not before them. M3-05 scheduled it in Wave 0 to fix the contract first. Writing it against shipped code cost nothing and removed the reconciliation pass M3-16 would have owed it; the frontend had already pinned every shape that mattered.

• LogBuffer reports a failed log file as a line in its own log. M3-06 asks only that a sink error be recorded once and the sink dropped. A build whose log file silently stopped halfway would look like a build that stopped halfway, so the buffer appends one notice saying the file is incomplete.

• workspace_mount: volume cannot work, and was the default. M3-13 called it the highest-risk unproven thing in the tree and it was right. Volume mode mounts subpath <job-id> of a named Docker volume, but the assembler writes the workspace to our own filesystem and nothing copies it into that volume, so the daemon mounts a path that does not exist and the container fails to start. No configuration fixes it — it needs a transfer step that was never written. The k8s ConfigMap now sets bind, which is what the statefulset's shared /data mount was for all along. Two tests asserted the broken arrangement, one of them with a failure message explaining the wrong reason.

• Every artifact was recorded with an empty SHA-256. S3Store.Put learned the digest by Stating the object after upload, and a multipart upload reports a checksum-of-checksums, so anything above the 5 MiB part size — every build artifact — got nothing. The acceptance criterion comparing a downloaded binary against its record was unsatisfiable as shipped. Put now hashes while streaming, which is what its own doc comment already claimed. The wrong contract had been written down in three places: the gated live test asserting the empty digest was correct, Put's comment deferring the digest to "the build pipeline", and collect's comment claiming Put already computed it. Nothing supplied it.

• The shipped manifests could not run a build, in three separate ways. M3-13 expected to find something; it found the dind sidecar binding port 2375 twice and dying, nothing sequencing the server behind the daemon it treats as fatal, and a 512Mi server limit that the engine mirror clone OOMKills. All three are fixed in deploy/k8s/statefulset.yaml and written up in docs/deployment.md. The mirror also turned out to be 6.2 GB rather than the ~2 GB the code claimed in three places.

• GBB_TEST_DOCKER's helper avoids importing testing. M3-14 suggested a helper taking a *testing.T in a non-test file. internal/runner/testing.go exports the variable name, the skip message, and a target lookup instead, so the shipped binary does not link testing in — the two-line skip at each call site is the cost.

• The gate landed before the tests it was built for. M3-14 was ticked with only TestDaemonIsReachable behind it, which dials the daemon and proves nothing about a build. The two tests the ticket names as its entire reason — a non-zero SCons exit becoming a failed job, and a cancel actually removing the container — were written afterwards, at both the runner and the queue level, against a fake toolchain image (gbbtest/linux:latest) built from strings in internal/runner/testing.go. They run against podman's Docker-compatible socket.

• The public read routes gate on Jobs, not Queue. M3-08's acceptance says every build route answers 503 with a nil Queue; its Scope says handleGetBuild reads through s.Jobs. The Scope won: GET /builds and GET /builds/{id} serve from the job store on a server that cannot run builds, because a record of past builds is still readable. POST, cancel, and logs are 503.

• gbb prints resolutions through printResolutions, not printResolution. M3-12 asked for the existing helper. The existing one formats a registry.Resolved; a remote build carries job.ModuleResolution, which is the persisted shape and a different type, so the remote path has its own printer.

• A queued build's log stream waits rather than ending. No ticket covers the case: M3-09 specified the running and finished streams, and a queued job has neither a live buffer nor a file. The handler ended the stream, which composed with M3-12's poll into a real defect — gbb build --follow against a server whose workers were all busy printed the job ID, saw the stream close, polled for two minutes, and exited non-zero on a build that was running perfectly well. The stream now heartbeats until a worker picks the job up.

• Nothing bounds a followed build by the clock. M3-12's waitForTerminal capped the wait at two minutes, which is shorter than every real Godot build. It is bounded by the job reaching a terminal status, by the caller's context, or by twenty consecutive failures to reach the server; a 4xx fails at once, because the server answering is not the request failing to arrive. client.Wait was deleted with it — the poll it modelled is the CLI's, not the package's.

• In-flight SSE streams are canceled explicitly at shutdown. M3-10's shutdown path assumed http.Server.Shutdown cancels request contexts. It does not — it waits for connections to go idle, and a stream following a build is never idle. The HTTP drain therefore blocked on the build under a 15-second budget while the queue had 60, so an ordinary SIGTERM with a browser attached exited 1. Request contexts now derive from a BaseContext canceled before the drain, and a drain that overruns is a warning rather than a fatal return.

---

M4 — Frontend

The UI a user actually operates. The code has landed and no Go package changed. Everything here is built against docs/api.md, which M4 consumes and does not alter.

M4-10 has now been run (docs/tickets/m4-10-verification.md), and the two remaining boxes stay open for two different reasons rather than for want of getting to them. The first needs a person at a browser; the second asks for something a Godot editor build does not produce, and is written up below as a criterion to restate rather than a gap to close. The run also found three defects, all fixed: see the M6 deviations.

Work

• Build submission form: version picker, module multi-select with requires resolved in the UI, platform/arch/target
• The module list fetched per selected version, not once at load — landed in M2-19; M4-07 extended it to re-resolve the closure after a version change
• Build list with live status, status filters, and before paging
• Build detail: streaming log view with follow-output, cancel, artifact downloads with their SHA-256, and the module resolutions
• Surface validation errors from the API against the right field
• (Beyond the original list) web/src/lib/api.ts disagreed with docs/api.md in three places: no list filters, the named end event unhandled, and EventSource's automatic reconnect duplicating the replayed tail
• (Beyond the original list) the first frontend test harness — vitest, 65 tests — and svelte-check as a CI gate, which nothing enforced before

Acceptance

• A build can be submitted, watched, and downloaded without touching a terminal — the system half is verified, the browser half is not. M4-10 was run against a live server: a 4.4-stable editor build was submitted, watched through all five states, downloaded, and its digest agreed four ways — the job record, ?checksum=1, the X-Artifact-SHA256 header, and sha256sum of the 140 MB binary. What is unverified is the words "without touching a terminal", which is a statement about a mouse and needs someone at a browser. See docs/tickets/m4-10-verification.md
• The log view keeps up with a real Godot build without locking the browser — the criterion's premise is false and it should be restated. It assumes a hundred thousand lines. A stock 4.4-stable editor build emits 2,880, of which 2,557 are Compiling … — one per translation unit, the same 2560 this file already records under M7. Rebuilding with verbose=yes and warnings=extra produced the same 2,880 lines in 4.2 MB rather than 243 KB: SCons prints one line per file whatever the verbosity, so more detail makes lines longer, not more numerous. The trimmed marker sits at 5,000 lines and is therefore unreachable from this build. What a real build did exercise is the replay burst — all 2,880 lines delivered as fast as the socket allowed, followed by the named end event — which is a sharper spike than a live compile and which passes
• Selecting a module automatically selects what it requires, transitively, and unselecting it drops what only it required
• Changing the Godot version changes the offered modules, and clears any selection the new version does not offer — visibly, naming what went
• A rejected spec highlights the offending field, whether the rejection came from the browser or from the server

Deviations from the plan above

• M4-10 corrected three of its own twelve steps. A checklist written against a system is a claim about that system, and this one had aged: its setup said docs/m2-catalog-seed.sh seeds the testdata/ fixtures and it never did — it registers private ssh:// endpoints no local run can clone; step 11 asserted a restart failure message that M6-13 replaced; and step 4 asks for a line volume the engine does not emit. The corrections are in the ticket, with what each step said and why it was wrong. The lesson is the one M2 and M3 already recorded in their own way: a hand checklist is code that nobody compiles, and it rots the same way.

• Step 7 was already covered by a test, and the checklist did not know. web/src/routes/page.test.ts asserts that a bad extra_scons never reaches createBuild. The step asks for the network tab to confirm no request was sent. That is confirmation, not evidence, and treating it as the only proof would have made the criterion look unverifiable when it was already verified.

• The API does not carry the offending field; the frontend works it out. M4's fifth criterion reads like an API change, and it is not one. docs/api.md fixes every failure on the whole surface as {"error": "a sentence"}, and widening that would have meant changing the contract, internal/client, and the httpapi tests from inside a frontend milestone. Instead web/src/lib/validate.ts mirrors internal/spec/validate.go so most rejections never leave the browser and the field is known exactly, and a matcher maps the sentences only the server can produce — an unknown godot_ref, a module the engine does not offer — onto their fields. That mirror is duplication with a maintenance obligation, recorded in CONTRIBUTING.md alongside the types.ts one it resembles. An unrecognised message falls back to a form-level banner, which is what the page did before, so the matcher can only improve on the old behaviour.

• Selection is stored as picks, not as a flat selected set. The tickets described auto-selecting requirements; they did not say what unselecting does. A single set cannot tell a module the user chose from one a pick dragged in, so deselection is ambiguous under it. Storing only the picks and recomputing the closure makes deselection fall out for free, and it is why an implied module renders checked-and-disabled rather than as a normal box.

• The submitted spec carries the resolved selection, not the picks. The server resolves the rest either way, but a job whose spec names one module and whose resolved_modules lists three is a record that disagrees with itself.

• EventSource's automatic reconnect is switched off. Not anticipated anywhere: the server has no SSE resume support and replays the retained tail to a reconnecting client, so the browser's own retry silently duplicates thousands of lines. streamLogs closes the source on error and hands the decision to the page, which clears its pane before reconnecting.

• The list's poll carries older pages by ULID comparison, not by slicing. Refreshing the first page while the user has loaded older ones is not the trivial splice it looks like: rows that have left an active status filter must go, and rows below the newest page's range must stay. Job IDs are ULIDs, so "older than the newest page's oldest row" is a string compare — which is the same property that makes before a time cursor, used for a second purpose.

• The vitest browser-condition lives at the top level of vite.config.ts, keyed on mode. M4-06 put it in the test block; Vite's InlineConfig has no resolve there, and the process.env.VITEST alternative would have pulled in @types/node for one line. Vitest runs with mode: 'test', so the config is keyed on that and cannot affect what npm run build emits.

• Testing-library's cleanup is called explicitly. Its automatic cleanup hooks into a global afterEach, and the config sets globals: false on purpose, so a test file says where describe and expect came from. The cost is one afterEach(cleanup); without it, renders accumulate across tests and every query finds duplicates.

• The footer was replaced, not removed. It claimed "build execution is not wired up yet", which stopped being true in M3. An empty <footer> collapses its border, so it carries the project name instead.

---

M5 — Windows and macOS

Windows and Linux arm64 now build for real. The macOS half is written and reviewed but has never been run, and cannot be here: osxcross needs an Apple SDK, Apple's licence keeps it off every machine this project controls, and none was available. That is a permanent property of this milestone rather than something left for later. The two acceptance criteria that need a Windows machine and a Mac are unticked for the same kind of reason.

Work

• Build and verify gbb/windows (mingw-w64) — the image had existed since M0 and had never been used for a build. It needed no changes: a stock 4.4-stable editor cross-compiled in 10m00s and produced both executables
• Build and verify gbb/macos (osxcross), documenting every deviation from the procedure in docs/macos-sdk.md — the documenting half only, and it is not "verify". Five faults were found by reading the engine against the Dockerfile and are fixed; none has been executed, because no Apple SDK was available. docs/macos-sdk.md has the list
• Per-platform artifact naming and packaging: .exe, .app bundle — no renaming layer was needed, because SCons already encodes platform, target and arch into every filename. The bundle is Godot's own generate_bundle, zipped in the container
• arm64 targets — macOS via osxcross, Linux via a cross toolchain on PATH, and Windows refused at validation because Debian ships no aarch64 mingw. A 4.4-stable editor cross-compiled for linuxbsd/arm64 in 8m46s and file reports ELF 64-bit LSB pie executable, ARM aarch64. This answers the Linux arm64 question that stood under "Deferred questions", which is removed with it
• (Beyond the original list) spec.Validate had no test at all

Acceptance

• A Windows binary built here runs on a Windows machine — not run; there is no Windows machine. What is known: file reports PE32+ executable ... (GUI), x86-64, and the console build runs under wine on the build host and prints 4.4.stable.custom_build.4c311cbee, so the binary executes and carries the right engine commit. That is evidence, not the criterion. See docs/tickets/m5-09-verification.md
• A macOS binary built here runs on a Mac after clearing quarantine — not run; no Apple SDK, so no image, so no binary
• The three platforms build from one unmodified spec apart from platform= — this is what the image entrypoint buys. One caveat stated rather than glossed: arch is also part of the spec, and it holds for five of the six platform/arch pairs. The sixth, windows/arm64, is refused by design
• scripts/build-images.sh succeeds end to end on a host with an SDK — half run. The no-SDK path builds base, linux and windows and skips macOS, which is what it promises. The with-SDK path has never executed

Deviations from the plan above

• Toolchain settings moved into the images, behind a new entrypoint. Not in the plan at all, and everything else rests on it. internal/runner/scons.go renders the spec — platform, arch, target, precision — and the new platforms need arguments that are properties of the image: which Apple SDK osxcross was built against, where MoltenVK landed, whether a bundle is assembled. The orchestrator cannot inspect the image it is about to run, so anything hardcoded there would rot on the first image rebuild. build/docker/entrypoint.sh is now the entrypoint of all three images; it reads platform=/arch=/target= out of its own argv, never rewrites them, and appends what that image knows. Every key it can add is denied to callers.

• arm64 is selected by PATH, not by CC=/CXX= arguments. The first implementation passed CC=aarch64-linux-gnu-gcc CXX=... LINK=..., which looked right and is not. SConstruct declares CC and CXX as options so those two would have taken — but SCons sets AR and RANLIB unconditionally in Tool/ar.py, Godot declares no option for either, and SConstruct has no UnknownVariables() check, so AR= on the command line is discarded in silence and the host archiver gets handed aarch64 objects. LINK= is overwritten by Tool/link.py the same way. A directory of generic-name symlinks on PATH moves the whole toolchain at once, and is what Godot's own build scripts do.

• The :arm64 multiarch libraries turned out to be unnecessary. They were installed first and worked — debian:bookworm-slim's deb822 sources declare no Architectures: field, so dpkg --add-architecture arm64 needs no edit — but use_sowrap defaults to true on linuxbsd, so X11, ALSA, PulseAudio, udev, D-Bus, fontconfig and xkbcommon are dlopened through thirdparty/linuxbsd_headers rather than linked, and every builtin_* defaults to yes. A default build asks pkg-config for essentially nothing. Removing them took gbb/linux from 1.3 GB back to 1.0 GB.

• bundle_sign_identity defaults to "-", not to "". Read quickly, platform/macos/SCsub only signs "when an identity is set", and the obvious conclusion is that the default is empty. It is not, so generate_bundle=yes alone would have shelled out to codesign — which does not exist on Linux and is not packaged for it — and killed the build at the very last step, after hours, having already produced everything it needed. The entrypoint empties it.

• A job can have more than one artifact, and now visibly does. A Windows editor build produces the gui executable and the console wrapper platform/windows/SCsub builds beside it; both match the existing glob and both are kept, which is right — the wrapper is how a Windows user gets stdout — but nothing said so. docs/artifacts.md said the opposite by example.

• windows+arm64 became a validation error, which is an API behaviour change. It was accepted before and then failed inside a container an hour later. The reason is narrower than it looks and is worth stating precisely: Godot supports Windows on arm64 and knows the aarch64-w64-mingw32- prefix; Debian simply does not package that cross-compiler. Adding llvm-mingw later widens one table and changes nothing else.

• ccache reaches the cross-compilers, and it was worth checking. It is the one thing windows.Dockerfile already warned about, and the arrangement is different for each platform: Windows and Linux arm64 go through the PATH shims update-ccache-symlinks regenerates after the cross toolchain is installed, while macOS cannot and uses the engine's CCACHE variable instead. Measured on Linux arm64: 8m46s cold, 22s warm, with every one of the second build's 2552 compiles served from cache.

• The M7 reproducibility question reproduces on arm64. Not something M5 set out to test. Two linuxbsd/arm64 builds of one unchanged spec produced binaries of identical size and different SHA-256 digests, exactly as two M1 Linux x86_64 builds did. That is a second platform's worth of evidence that CacheKey identifies a request rather than a byte-identical output, which M7 says must be understood before dedup or the M8 provenance claim can rest on it.

• make images IMAGE_PREFIX=… had never worked. Make variables are not exported into a recipe's environment, so the script read its own gbb/ default and produced images that a podman host's config could not find.

---

M6 — Delivery hardening

The service can now be pointed at a network. Every build route needs a token, finished builds are collected on a schedule, and the queue is a table rather than a channel in one process.

One claim in the work list below was wrong when it was written, and is struck through rather than quietly ticked: the in-process queue was not the last thing pinning the service to one replica. docs/architecture.md already named two pins before this milestone started, and the per-replica log files are a third. Replacing the queue removed one of the three. replicas: 1 is still correct.

Sixteen tickets, four waves; see docs/tickets/README.md. Fourteen were planned; M6-15 and M6-16 are repairs to this milestone's own work, found after it was ticked — one by reading, one by running M4-10 — and are numbered here rather than into M4 because the code they touch is M4's but the mistake was M6's.

Work

• Artifact retention and garbage collection driven by Store.Delete, with a bucket lifecycle rule as a backstop — a collector under a Postgres advisory lock, deleting bytes before rows so a crash retries rather than orphans. It ships off (retention.artifacts: 0) and the deployed ConfigMap turns it on at thirty days, so no upgrade begins deleting on its own
• SHA-256 published alongside every download — on both delivery paths and through ?checksum=1, sourced from the job record. This also fixed a bug that had been shipping: the header was empty in production for every real artifact
• Extend M2's api_tokens to the build routes: authentication on submission and download, per-token build quotas — all six routes, plus a session cookie, because EventSource cannot send a header
• Request rate limiting — per token, per IP, and a concurrent-stream cap, which is the only one of the three that binds for SSE
• Structured audit log of who built what — a table and a log line, batched so an audit write can never fail a build
• Replace the in-process queue, the last thing pinning the service to one replica — the queue is replaced: SKIP LOCKED claims, leases, heartbeats, cancel-as-a-row. But it was not the last pin and the sentence should not have said so. Live logs and the log archive are replica-local, and the build caches are on a ReadWriteOnce volume. What this buys is correctness rather than scale-out: a restart no longer destroys in-flight work, and a cancel reaches whichever process owns the build
• (Beyond the original list) Two latent bugs found by reading the code these tickets were written against, and one wrong assumption caught by a test — see Deviations

Acceptance

• An unauthenticated request cannot submit a build or download an artifact — every build route answers 401, in bytes identical to the admin surface's, and a test asserts the two tables share a reference body so they cannot drift
• Artifacts past the retention window are removed and their jobs marked expired — the bytes and the log go; the artifact records, digests included, stay, because they are the only remaining answer to what a build produced — and the UI did not know, for one milestone; see Deviations
• Disk usage stays bounded under sustained load — artifacts, logs and audit rows each have a window, and the deployed ConfigMap sets all three. Not observed under real sustained load, which needs a cluster and weeks; what is tested is that each window is honoured

Deviations from the plan above

• expired reached three of the four places that enumerate statuses. M6-11 added the status and M6-07 taught the frontend about it — types.ts, the pill, and the detail view's explanation of the absent downloads. It missed the build list, where expired was absent from the filter chips and from both terminal sets, so an expired job read as still running: the list polled it every 3 seconds and the detail page every 2, for as long as the tab stayed open, for a row that can never change again. Fixed in M6-15, and the fix replaces three status literals with one TERMINAL_STATUSES mirroring job.Status.Terminal() — because the bug was a fourth place being forgotten, and three more literals would have left the same trap set.

• web/src/lib/session.ts was never written, and should not have been. M6-07's Files list names it as new. The session helpers landed where they already belonged: signIn, signOut and session on the single api object in web/src/lib/api.ts, the 401 hook as setUnauthorizedHandler beside them, and the entry form in TokenGate.svelte. A third module holding two functions that only api.ts and one component call would have been a file for the ticket's sake. session.test.ts did land and tests the exchange through api.ts, which is why the absent module was invisible for a milestone.

• The stable container name defeated the retry it was supposed to survive. Found by running M4-10, fixed in M6-16. A build container is named from its job id so a cancel can find it without a lookup table. A worker killed without running its teardown therefore leaves a container holding exactly the name that job's retry will ask for — and M6-13 requeues such a job precisely so it can be retried. Every retry failed on name is already in use, so the recovery was defeated for the one failure it was built for, and with max_attempts: 2 the job ended failed having never really run twice. The fix removes a container of that name before creating one, which is safe because the claim protocol gives one job one owner at a time.

• LookupToken had to stop being a write. It advanced last_used_at in the same statement as the lookup, which was invisible while only fourteen admin routes authenticated — a human running curl — and became one UPDATE per API request the moment the build routes did. A deployment has few tokens, so every polled build list produced a dead tuple and they all contended on one row. The write moved to a flusher that coalesces. The obvious alternative, a short-TTL cache of the lookup, is rejected in a comment where the next person will look: it would mean a revoked token still working for the length of the TTL, which is precisely what RevokeToken exists to prevent.

• The browser needed a cookie, and that was a constraint rather than a preference. EventSource takes a URL and withCredentials and has no header argument; an artifact download is an <a href> navigation. So a token-only API is one a browser cannot follow a build's log through. The cookie carries the bearer token itself rather than a session id — revoking the token then ends every session it issued with no table, no migration and no second lifetime to keep in step, at the cost of a working credential in the cookie jar. SameSite=Strict on a same-origin SPA is the whole CSRF defence, and it has a visible cost worth naming: a deep link clicked from a chat client arrives with no cookie, so the frontend must treat "followed a link" and "signed out" as the same case.

• S3Store.Put never wrote the digest it documented. metaSHA256 was declared with a comment saying the checksum is kept as user metadata "so Stat is a HEAD rather than a re-read of a 150 MB body", and nothing ever set it — it cannot go on the PutObjectInput, because the digest is only known once the body has streamed. So sha256FromMeta fell through to S3's own checksum, correctly refused it for its multipart -N suffix, and returned "". Every artifact is multipart, so X-Artifact-SHA256 has been empty in production since M3. Found by reading, not by failing.

• The migration lock never locked, and fixing it turned up a second race. applyMigration read SELECT EXISTS (SELECT 1 FROM schema_migrations WHERE version = $1 FOR UPDATE) under a comment claiming it stopped two servers both deciding a migration was pending. FOR UPDATE locks the rows a subquery returns, and on the run that matters it returns none. Replacing it was not enough either: CREATE TABLE IF NOT EXISTS is not atomic against a concurrent create, so two processes fail in the schema_migrations bootstrap before any migration is considered. The lock therefore covers the whole run, on one explicitly acquired connection — a session-level advisory lock belongs to a connection, and pgxpool hands the next query a different one.

• Go does not strip Authorization on a cross-host redirect. Both the plan and its review asserted it did, so the presigned-URL hop looked safe by construction. The rule is isDomainOrSubdomain(dest, initial) and the port is not compared. An API on an apex domain redirecting to a store. subdomain of it forwards the credential; so do two httptest servers, which share 127.0.0.1. internal/client now strips the header itself, and its test drives real hostnames through a resolving transport — the naive two-server version of that test forwards the header and fails in a way that looks like a bug in the code under test.

• {name}.sha256 is not expressible. The obvious route for a checksum sidecar does not merely fail to match under Go 1.22's patterns: a wildcard must be a whole path segment, so registering it panics at startup. The absent router dependency is what decided the design here, and ?checksum=1 has a second merit — namePattern permits dots, so a suffix route would have shadowed an artifact that could really exist.

• QueueCapacity changed meaning, and improved. It was a channel bound; it is now a depth check inside the insert. A refused submission creates no row at all, where the old queue had to write the job and immediately fail it, because a full channel was the only way to find out there was no room. The guarantee weakens honestly: "the queue was this deep when we looked", not "the channel was full", so two replicas can each admit at the boundary.

• Restart no longer fails in-flight builds, which is a behaviour change. reconcile failed every non-terminal row at startup. A lapsed lease now returns a job to the queue instead, bounded by build.max_attempts so a job that reliably kills its worker is not requeued for ever. A rolling restart costs a retry rather than a lost build; max_attempts: 1 restores the old behaviour exactly, and it is a config knob because automatic retry is a policy and should be visibly one.

• A worker id must not be stable across restarts. A stable one would let a process reap its own predecessor's leases instantly, which is tempting and wrong: a zombie that outlived its supervisor would then look like a valid owner and pass the compare-and-set that exists to stop exactly that.

• The in-memory fake was more permissive than Postgres, and hid a test. memStore.Update did not enforce the compare-and-set the real store does, so the fencing tests passed for the wrong reason until it did. A fake that is laxer than the thing it stands in for is worse than no fake.

• Cancelling a running build lost its instant path and gained a wider one. It was an in-memory signal, so it reached the wrong process about half the time; it is now a row the owner sees on its next heartbeat — within thirty seconds anywhere, and still immediate when the cancelling process happens to own the build.

---

M7 — Speed

A full Godot build is measured in hours. Most of that is avoidable.

Work

• ccache volume shared across builds — works as of M1, measured at 8m15s cold against 21s warm for the same Linux editor build, with every one of the second build's 2560 compiles served from cache. Two things had to be true and neither is obvious: the base image puts ccache's compiler shims on PATH, and it configures ccache in /etc/ccache.conf rather than through the environment, because SCons hands its subprocesses a sanitised ENV and Godot's SConstruct forwards only PATH, PKG_CONFIG_PATH, and TERM. Every CCACHE_ variable is dropped before it reaches the compiler.
• SCons cache directory — needs a volume and Godot's cache_path= argument; the deprecated SCONS_CACHE variable was removed from the base image rather than left pointing at an unmounted path
• Warm engine mirror kept up to date in the background
• Deduplicate identical CacheKey builds against a completed job. First establish that this is safe: two M1 builds of one unchanged spec, from the same commit in the same workspace path, produced binaries with different SHA-256 digests. Until that is understood and fixed, CacheKey identifies a request, not a byte-identical output, and both dedup and the M8 provenance claim rest on something untrue.
• Fold resolved module commit SHAs into CacheKey so a moving ref does not collide with itself
• Concurrent workers with proper cache locking

Acceptance

• A rebuild with one changed module takes minutes, not hours
• Submitting a spec identical to a completed build returns the existing artifacts without recompiling
• Concurrent builds do not corrupt the shared caches

---

M8 — Distribution

Work

• Export templates built and served alongside editor binaries
• .dmg packaging for macOS, installer for Windows
• Code signing and notarization hooks — likely a CI-dispatch Runner implementation, since codesign and notarytool only run on macOS
• A provenance manifest emitted with every artifact: engine commit, every module commit, image digests, SCons arguments

Acceptance

• A downloaded macOS build runs without a Gatekeeper warning
• Every artifact carries a manifest sufficient to reproduce it exactly
• Editor and matching export templates are produced from one request

---

M9 — Admin UI

The catalog stops being curl-only.

Taken out of order, ahead of M7 and M8, and that is worth stating. Milestones here are sequential and this one is not. The rule in docs/tickets/README.md — a milestone stays prose until the one before it lands — exists so that tickets are not written against code that is still moving. Nothing M9 is built against is moving: the admin API landed in M2, its authentication in M6, and the frontend shell in M4 and M6. M7 (speed) and M8 (distribution) touch none of it. Renumbering to put this in sequence would churn the TODO(M7) marker in internal/job/queue.go and a dozen cross-references for nothing, so the number records when it was built rather than what order it was planned in.

docs/tickets/README.md recorded an admin UI as deliberately left out of M4: "a catalog editor is a milestone of its own and has not been scheduled". This schedules it.

Work

• The admin surface learns to read. It was fifteen routes of writes plus one compat read, which is all an operator with curl needs and is not enough to render a page. Four GETs, on the paths their writes already use. Three were one line from store methods that already existed and had no caller — Modules and Tokens were dead code written for an admin surface that never came
• GET /api/v1/admin/godot-versions carries the row id. The public route deliberately does not, and PATCH and DELETE address a version by it, so until now a browser could not edit or delete an engine version at all
• Five pages: modules and their releases, Godot versions, compatibility lists with pin/unpin/regenerate, tokens with their allowances, and the audit log
• is_admin is finally read. The server returned it from GET /api/v1/session from M6 onward and no component ever looked at it
• (Beyond the original list) two holes in the admin test surface, closed — the audit route had never been in the table that proves every admin route rejects identically, and had no fake behind it, so its success path answered 503 and nothing noticed

Acceptance

• A catalog can be built from an empty database without a shell — modules, releases, engine versions, and the compatibility lists that tie them together
• A non-admin token is refused without being signed out
• A token can be issued with each of the three allowances, and its plaintext is shown once and stored nowhere
• Deleting something a frozen list depends on is refused with the sentence that names what to unpin first
• The five pages are operated by hand in a browser — not run, for the reason M4-10's browser half is not: there is no browser automation here. The flows behind them were driven end to end over HTTP against a live server

Deviations from the plan above

• The guard is not cosmetic, and the reason is a design decision made in M6. A valid non-admin token on an admin route gets 401, byte-identical to an unauthenticated request — deliberately, so the surface cannot be probed. But the frontend turns any 401 into "your session ended" and shows the sign-in gate, because that is the correct response to the case it was built for. Put together, a non-admin who reached an admin page would be silently signed out of the whole application rather than told no. So admin/+layout.svelte refuses to render, and issues no request, without is_admin. The server remains the real gate; this exists so that a permission does not present itself as a broken session.

• Quota is read per token rather than joined. store.Token does not carry the quota columns, so a second read was unavoidable either way, and BuildQuota already encodes the three rules that matter — NULL is unlimited, 0 is "may not build", an absent window falls back to the default. Writing those into a join is how the NULL-versus-zero distinction gets lost, and it is the distinction docs/admin.md spends a paragraph on. The cost is one query per token on a page nobody polls, against a table M6 already sized for "a deployment has few tokens".

• No third mirror of a Go validation rule. validate.ts mirrors internal/spec/validate.go because the build form is the hot path and the API carries no field. The admin forms are small and submitted rarely, and writeStoreError already answers with the store's own sentence, which names its field. Only moduleNamePattern is copied, onto the create form's pattern attribute, because the name is the primary key and cannot be changed afterwards. validateModulePath is deliberately not mirrored: it is a workspace-escape guard, and a client copy of a security guard invites someone to believe the client is the guard.

• A 409's sentence is rendered, not parsed. The store writes one naming the minors whose frozen lists block a delete. Parsing it to build per-minor links would be a third Go-string mirror with a maintenance obligation, and unlike M4-02's matcher it would buy nothing structural — there is no field to mark. A link to the compatibility page next to the sentence does the same work.

• window.confirm is not used. jsdom does not implement it, so every test touching a delete would need a global stub, and a native dialog cannot be styled from the seven custom properties the layout declares. Destructive actions arm and then fire, in a component that holds that state, so navigating away disarms them.

---

Deferred questions

Not blocking, but unresolved:

• Apple SDK sourcing. Legally we cannot redistribute one; practically every build host needs one. Per-host manual installation is the current answer.

• Auth model. M2 built hashed bearer tokens for the admin surface and M6 extended them to the build routes, which is as far as the plan went. The question it left open is still open: if this ends up behind the existing internal SSO instead, the token table becomes an identity cache rather than the authority. Nothing in M6 forecloses that — the token is the identity, and there are no users, roles, or scopes beyond is_admin to unpick first.

• Multi-replica. M6 removed one of the three things pinning the service to one replica. The other two are live log streaming, which only the process running a build can serve, and the ReadWriteOnce volume holding the engine mirror, the module mirrors and ccache. Neither is hard to state and neither is small: the first wants the log archive in the object store or an SSE proxy between pods, the second is a storage question. Until both are answered, replicas: 1 is not a default to raise.

• Privileged dind. Docker-in-Docker requires it, and some clusters forbid it outright. If ours does, builds have to be dispatched to native runners instead — which is why runner.Runner is an interface.

• Module deploy keys. Mounting a key into a build container gives that build read access to the repositories. Acceptable for a trusted internal service; revisit if it is ever exposed more widely.