Artifact storage and delivery

Finished builds go to an S3-compatible object store at https://store.picklemustard.dev. They are not kept on the pod's volume.

Why the object store

A Godot editor binary is 50-150 MB, and a build can produce several. Keeping them on the pod's PersistentVolumeClaim would mean:

• artifacts die with the volume,
• the volume has to be sized for the whole retention window rather than for in-flight builds, and
• a second replica would not see the first one's artifacts.

Moving them out solves all three. It also removes most of what pins the service to one replica — what remains is the git caches on the local volume, not the artifacts. The catalog and the jobs live in Postgres, which several replicas could share.

Layout

godot-builds/                          bucket
└── artifacts/                         artifacts.s3.prefix
    └── 01J8ZQ.../                     job ID (ULID)
        ├── godot.windows.editor.x86_64.exe
        └── godot.windows.editor.x86_64.console.exe

A job usually has more than one artifact. A Windows editor build produces two executables — the gui one and the console wrapper that platform/windows/SCsub builds beside it, which is how a Windows user gets stdout. A macOS editor build produces the raw binary and a godot.macos.editor.<arch>.app.zip: the .app is assembled by Godot's own generate_bundle, and zipped inside the container because a bundle is a directory, collect carries only regular files, and an artifact name cannot contain a slash.

The prefix keeps one bucket usable for more than this service. Both backends derive keys from the same function, so artifacts can be migrated between local disk and the object store by copying keys verbatim — there is a test pinning that layout.

Delivery: presigned redirect

GET /api/v1/builds/{id}/artifacts/{name} answers 302 with a presigned URL, and the client fetches the bytes from the object store directly.

The alternative — streaming through the API server — costs its bandwidth and holds a connection open for the length of every transfer, and buys nothing the object store does not already do. It stays available for deployments whose clients cannot reach the object store:

artifacts:
  proxy_downloads: true

The filesystem backend cannot presign, so it always streams. That is what artifact.Presigner is for: the handler redirects when the store implements it and streams when it does not.

The handler HEADs the object before redirecting, so a missing artifact is a 404 from us rather than an opaque error from the object store after the client has already followed a redirect.

artifacts.download_ttl (default 6h) bounds how long a link keeps working. Long enough to finish a large download on a slow connection; short enough that a leaked URL expires.

Credentials

From the environment, never the config file, so they can be a Kubernetes Secret:

Variable	Meaning
GBB_S3_ACCESS_KEY_ID	Access key
GBB_S3_SECRET_ACCESS_KEY	Secret key

deploy/k8s/secret.yaml is a placeholder with REPLACE_ME values. Replace it with a Secret created out of band, or one from whatever secret manager the cluster uses. Leaving both variables unset falls back to the default AWS credential chain, which is preferable wherever the platform provides one.

Path-style addressing

artifacts.s3.use_path_style defaults to true, giving endpoint/bucket/key rather than bucket.endpoint/key. Virtual-host addressing needs a wildcard DNS record and a certificate covering it, which self-hosted S3 implementations usually lack. If store.picklemustard.dev does have wildcard DNS, turning this off is slightly cheaper on the routing path, but path-style works either way.

Checksums

Ref.SHA256 is the object's digest, and the upload asks the store to verify a SHA-256 as it receives the bytes, so a corrupted transfer fails rather than being stored.

One caveat: for a multipart upload S3 reports a checksum-of-checksums with a -N part-count suffix, which is not the file's digest and must not be presented as one. sha256FromMeta refuses those and falls back to user metadata. Since a build artifact is always large enough to go multipart, the store computes the digest itself: S3Store.Put tees the body into a sha256.Hash as it uploads, and that is what lands on Ref.SHA256 and on the job. If S3 does report a usable digest — a small enough object to go single-part — and it disagrees with the computed one, Put fails rather than recording either.

This paragraph described an intention rather than the code until the M3-13 cluster run: Put used to read the digest back with a Stat after uploading, get a checksum-of-checksums, discard it, and return an empty SHA256, on the understanding that something further up would fill it in. Nothing did. Every artifact was stored with no digest at all.

It described the stored object wrongly for longer still. metaSHA256 was declared with a comment saying the checksum is kept as user metadata "so Stat is a HEAD rather than a re-read of a 150 MB body", and nothing ever wrote it — it cannot go on the PutObjectInput, because the digest is only known once the body has streamed. So sha256FromMeta fell through to S3's own ChecksumSHA256, correctly refused it for its multipart -N suffix, and returned "". Every Stat and Open of a real artifact reported no digest, and the X-Artifact-SHA256 header was empty in production. M6-12 writes the metadata with a same-bucket CopyObject — server-side, no bytes over the wire — immediately after the upload, and logs rather than fails if that annotation does not land, because the job record is the copy that matters.

List still returns Refs with no digest, and now says so on the interface: an object listing carries no user metadata, so filling it in would mean a HEAD per key for a value its only caller does not use.

That is the digest a client must verify against, and it is what the download endpoint publishes — not the store's view of it, which may be absent. Three ways to reach it, all carrying the same value:

• job.artifacts[].sha256, which is the contract and is what gbb uses;

• X-Artifact-SHA256, set on the proxied response and on the 302 itself. A Go client can read the redirect's header through http.Client.CheckRedirect, whose req.Response is that response; a browser cannot see it at all;

• GET /api/v1/builds/{id}/artifacts/{name}?checksum=1, which returns <hex> <name> — what sha256sum -c reads:

  curl -fsSLO -H "Authorization: Bearer $GBB_TOKEN" "$GBB/api/v1/builds/$ID/artifacts/$NAME"
  curl -fsS  -H "Authorization: Bearer $GBB_TOKEN" "$GBB/api/v1/builds/$ID/artifacts/$NAME?checksum=1" | sha256sum -c
  

  A query parameter rather than a {name}.sha256 path, because the path form cannot be expressed: Go 1.22's patterns require a wildcard to be a whole path segment, and registering {name}.sha256 panics at startup. It also avoids shadowing a real artifact, since a name may contain dots.

gbb fetch and gbb build both compute the SHA-256 of what they wrote and compare it to the Ref on the job, exiting non-zero on a mismatch; internal/client.Download returns it so any other caller can do the same. The comparison is exercised by tests on both sides, and the round trip has now been run against store.picklemustard.dev: a 40 MiB artifact uploaded in eight parts (ETag suffix -8), fetched through the API's 302, and its SHA-256 compared to the record. See the M3-13 section of deployment.md.

Retention

The service deletes; the bucket catches what it missed. Both halves, as this section always said they should be: a lifecycle rule alone leaves the database referencing objects that are gone, so Store.Delete is driven by the service and keeps the two in step.

A succeeded build's artifacts are collected retention.artifacts after it finishes. The deadline is stamped on the job at the moment it succeeds, not computed when the collector runs, so shortening the window later never retroactively deletes artifacts someone was told would live longer. Its job then reads expired — a terminal status that is not a failure. The build succeeded; only the bytes and the log are gone.

The artifact records are kept. Name, size and SHA-256 stay on the job, because they are the only remaining answer to "what did this build produce". Downloading one is 410 Gone rather than 404, which is the difference between "you got the name wrong" and "this is how retention works".

retention.artifacts defaults to zero, meaning keep forever, so upgrading an existing instance never silently begins deleting. deploy/k8s/configmap.yaml sets thirty days explicitly — that is what makes "disk usage stays bounded under sustained load" true of the deployed service, in a file where an operator can see exactly what will be removed and when.

One pass runs under a Postgres advisory lock, so exactly one replica collects however many are running. It deletes the bytes before it moves the row: a crash between the two leaves a job whose objects are gone, which the next pass retries and fixes because the deadline is still due — where the other order would leave an object no row points at, which nothing ever finds again. A failed delete leaves the job unexpired for the same reason.

Log files are pruned in the same pass but outside that lock, on retention.logs, keyed off each file's own mtime. They are replica-local on the ReadWriteOnce volume, so under the lock only the lock holder would ever prune its own disk; and mtime is used rather than a job's deadline because a failed build has no deadline, and its log is both the one worth keeping longest and the one that accumulates.

The bucket lifecycle backstop

deploy/s3/lifecycle.json, applied by hand:

aws s3api put-bucket-lifecycle-configuration \
  --bucket godot-builds --lifecycle-configuration file://deploy/s3/lifecycle.json

Not code. Setting it needs s3:PutLifecycleConfiguration, a bucket-administration permission far larger than the Get/Put/Delete on one prefix the server holds today — granting a service the ability to rewrite its bucket's lifecycle so it can set one rule is a bad trade, and a self-hosted implementation may not support the call at all. Not prose either: the rule has to be exact, and a paragraph rots.

Two rules. The prefix expiry is set longer than retention.artifacts on purpose: the service does the deleting and the bucket only catches what it missed, so a rule that fired first would delete objects the database still believes in — exactly the failure this design avoids.

The second rule earns the file on its own. AbortIncompleteMultipartUpload after seven days: every artifact is uploaded multipart, and an aborted 150 MB upload is billed storage that appears in no listing and that Store.Delete will never find, because there is no object to delete.