feat/distcc-and-crawl-tables #10

Merged
picklemustard merged 3 commits from feat/distcc-and-crawl-tables into main 2026-09-17 23:23:48 +00:00
No description provided.
Every release of every module came back needing review. Reproduced against
limboai, the one module in the deployed catalog: its README carries exactly the
compatibility table this scanner exists to read, and four separate defects each
stopped it dead.

  - scanTables required /godot|engine/ in a *header* cell. LimboAI's headers are
    "Plugin Version", "GDExtension/AssetLib" and "Module"; the word is in the
    cells. godotCol stayed -1 and the whole table was skipped.
  - versionCellMatches trimmed decoration from the ends only, so "`1.8.x`
    releases" kept a trailing word that defeated both the equality test and the
    .x suffix test. A range of the module's own releases was not handled at all.
  - boundsFromCell's patterns were anchored on a bare number, so "Godot 4.7",
    "Godot 4.6 or higher" and "Godot 4.4, 4.5, 4.6" all failed.
  - the prose fallback was defeated by markdown: engineNum's separator class is
    narrow enough that the "**" in ">**Supported Godot Engine:** **4.6**" breaks
    it, and floorPat's `supports?` cannot match "Supported" -- the \s+ after it
    lands on the "e".

A column may now be identified by its cells when no header names the engine.
That does not relax the package rule: every cell read this way contains the
literal word, which is the same guarantee the header requirement was providing.

Two changes here are more than widened patterns.

A series row now applies rather than proposing, unless the same table names the
release exactly. Most support tables are written entirely in series rows -- the
whole of LimboAI's is -- so treating every one as a proposal left every release
of such a module in review, which is the failure this commit is about.
Specificity still decides where the two meet: TestScanSupportTableSeriesYieldsTo
AnExactRow covers the half the old TestScanSupportTable case was pinning.

A table carrying two engine columns no longer deadlocks. LimboAI states both a
GDExtension and a Module engine and they disagree by a minor; read naively both
are exact, Resolve's contradiction check fires, and the candidate lands back in
review with a different note. gbb compiles a module into the engine, so the
Module column settles it and the other is demoted to a proposal rather than
dropped -- a reviewer can still see that the repository also says 4.6.

repoEvidenceFor was separately broken, and its own comment described the
behaviour it did not have: it re-scoped the default branch's evidence onto each
tag rather than rescanning it. Scan compares a table's version column against
the tag it is given, so the single scan with no tag in hand matched no row and
a README at HEAD contributed nothing to any release. The documents are held and
scanned per tag now; blobReader caches by object id, so this is CPU and no
further git IO, and TestModuleVersionsReadsASupportTableAtHEAD asserts both.

The fixture is limboai's real README, taken from the mirror the deployed service
keeps. Four things had to fail together to produce the symptom, so one real
document is worth more here than four synthetic cases.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
feat(runner): distribute compilation across a distcc pool
All checks were successful
CI / Dedupe (push) Successful in 10s
CI / Go (push) Successful in 2m55s
CI / Frontend (push) Successful in 29s
CI / Badges (push) Has been skipped
CI / Publish images (push) Has been skipped
CI / Dedupe (pull_request) Successful in 1s
CI / Go (pull_request) Successful in 1m34s
CI / Frontend (pull_request) Successful in 29s
CI / Badges (pull_request) Has been skipped
CI / Publish images (pull_request) Has been skipped
698cbaa232
A build gets four cores inside the dind sidecar, on whichever node gbb-0
happens to sit on; the cluster has forty-five across three. deploy/k8s/distccd
puts one compile server on each node so a build can use them. The sidecar is
unchanged and still runs the build -- it holds the workspace, the ccache volume,
the preprocessing and the link -- and only the per-translation-unit compile goes
out. The pool is never on the critical path: one that is gone or unresolvable is
a slow build, not a failed one, and the server says so in the build log.

Off by default, because enabling it trades a stated guarantee rather than
refining one. A build container otherwise runs with NetworkMode "none", and the
distcc client lives inside the compiler driver and opens the socket itself, so
there is no arrangement that distributes work and keeps the isolation. The
network follows the resolved host list and nothing else, so it stays denied for
a disabled deployment, for macOS -- the pool carries no osxcross -- and for a
pool that did not resolve. docker_test.go pins that the two halves cannot
disagree: a container granted the network but handed no hosts has given the
guarantee away for free, and one handed hosts but denied the network fails on
its first object file with a connection error nothing attributes to the network
mode.

The pool runs an image this repository builds, and that is the point rather than
an inconvenience. distcc splits one compilation across two machines and the two
halves must be the same compiler; the DaemonSet this replaces ran a public image
on Debian 13 (gcc 14.2) against these toolchains' bookworm (gcc 12.2), which
distcc says plainly produces objects that are wrong in ways nobody traces back
to the build farm. distccd.Dockerfile derives from gbb/base and adds the mingw
and aarch64 cross toolchains, because a cross-compile is distributed exactly
like a native one: the client sends the *basename* of its compiler and the
server resolves it in its own PATH. scripts/release.sh pins its tag alongside
the other three, and deploy/k8s/README.md says what follows from that -- a
release that rebuilds the toolchains must roll this DaemonSet with it.

Three details are not the obvious arrangement, and each was established by
running the thing rather than by reading about it.

  prefix_command in /etc/ccache.conf, not a masquerade directory on PATH. The
  obvious /usr/lib/ccache then /usr/lib/distcc does not terminate: each shim
  finds the real compiler by re-searching PATH minus its own directory, so they
  find each other and ccache exits with "recursive invocation" on the first
  object. prefix_command has no such ambiguity -- ccache has already resolved
  the compiler to an absolute path -- and it puts the cache in front of the
  network, so a hit never leaves the container.

  prefix_command points at a wrapper, not at distcc. distcc keeps its lock
  directory under $DISTCC_DIR, falling back to $HOME/.distcc, and reads both
  from the environment with no getpwuid fallback of the kind ccache has. Under
  Godot's SConstruct, which forwards only PATH, PKG_CONFIG_PATH and TERM,
  neither is ever set: distcc warns that HOME is unset, fails to take a lock,
  and compiles every file locally. The build succeeds. The only symptom is that
  distributing it changed nothing.

  The host list travels in a file. DISTCC_HOSTS is stripped for the same reason,
  so the runner hands the container GBB_DISTCC_HOSTS -- named for this service
  because spelling it DISTCC_HOSTS would imply a forwarding that does not
  happen -- and the entrypoint writes /etc/distcc/hosts, the one location distcc
  consults that does not depend on a variable. Both that file and
  /etc/ccache.conf are group-writable by the uid that builds, the same
  arrangement and the same reason as /ccache.

arm64 needs one thing more, and it fails silently without it. The cross
toolchain is on PATH under generic names because AR and RANLIB cannot be set
from the scons command line, so the name that would travel is "gcc" -- and the
server would answer with its native x86_64 compiler, hand back x86_64 objects,
and let the link fail hours later with no mention of distcc. The entrypoint
passes CC and CXX by triple for that case, which SConstruct does declare as
options. Uppercase keys were already unreachable through extra_scons; that is
now pinned by a test rather than left as a property of a regexp.

The service is resolved from a headless Service rather than a ClusterIP. A
virtual IP would hand each connection to whichever backend kube-proxy picked, so
distcc would see one host, could not account for its slots, and could not take a
failing server out of rotation. Resolution is net.LookupHost at build start: no
client-go, and no RBAC in a second namespace.

deploy/k8s/distccd is a kustomization of its own because the base sets
`namespace: gbb` and that transformer rewrites Namespace objects themselves,
so a second namespace cannot be expressed there. The local overlay keeps distcc
off -- a k3d cluster is several containers on one laptop, so a pool across its
nodes would distribute work to the cores that sent it -- and a test now loads
that overlay's config.yaml and asserts it, because the patch replaces the key
wholesale and inherits nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Merge origin/main (v0.6.1) into feat/distcc-and-crawl-tables
All checks were successful
CI / Dedupe (push) Successful in 1s
CI / Dedupe (pull_request) Successful in 1s
CI / Go (push) Successful in 1m35s
CI / Frontend (push) Successful in 29s
CI / Go (pull_request) Successful in 1m34s
CI / Frontend (pull_request) Successful in 32s
CI / Badges (push) Has been skipped
CI / Publish images (push) Has been skipped
CI / Badges (pull_request) Has been skipped
CI / Publish images (pull_request) Has been skipped
c5319d6568
The release bumped gbb-server and gbb-gateway in kustomization.yaml and
build.image_tag in the ConfigMap; this branch added a note to the same images
block and its own settings further down the same ConfigMap. Both version bumps
are taken.

deploy/k8s/distccd/kustomization.yaml is bumped to v0.6.1 by hand, because it
did not exist when the release ran and scripts/release.sh could not rewrite it.
That file is the one tag that must not lag: distcc runs the client half of a
compile in the toolchain image and the server half in the pool's, and two
different compilers across that seam produce objects nothing reports as wrong.
All four references -- server, gateway, build.image_tag and the pool -- now name
v0.6.1.

The header comment in kustomization.yaml claimed it was the only file carrying a
tag, which stopped being true on this branch. Corrected rather than left to be
read as still complete.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
StudioTeal/GodotBinaryBuilder!10
No description provided.