Kubernetes phase execution: first implementation slice

internal/execution/kubernetes is an execution adapter using the Kubernetes Go client. It is not wired into the API queue and does not replace the deployed Docker runner yet. Local CLI compilation remains available.

An immutable Plan identifies a build by ULID and attempt (1 or 2), namespace, dynamic storage class, fixed deadline, digest-pinned phase images, trusted preparation/publication entrypoints, and the effective compilation spec. The module list must contain the actual grafted directory names, including resolved dependencies. Public requests must never supply phase commands.

The adapter creates a 20-GiB RWO PVC and deterministic preparation, compilation, and publication Jobs. Ensure attaches to existing resources and verifies their plan hash and recorded UID. A missing resource with a recorded UID is a conflict, not permission to start another build. Concurrent ensures converge on one Job.

Compilation runs scons -j4 in /workspace/engine, requests and limits four CPUs and eight GiB, and has a deny-all ingress/egress NetworkPolicy. Preparation runs in /workspace and must create the engine directory. Publication mounts the workspace read-only. All phases run as UID/GID 1000, without an automatically mounted service-account token, privilege escalation, host mounts or capabilities. Root filesystems are read-only; bounded /tmp storage is writable. Phase images must support these settings. No credential volumes or environment variables are provided by this slice.

Jobs have no automatic retries (backoffLimit: 0), use the Failed pod replacement policy, and receive the remaining attempt duration as their active deadline. Before advancing, the adapter requires successful predecessor Jobs and no pending, running, unknown, or terminating predecessor pods. Cancel suspends existing Jobs and waits for pods to stop. Resources are retained until an explicit Cleanup after durable outcome/log persistence. Cleanup uses UID preconditions and foreground Job deletion before deleting the workspace and policy.

Kubernetes Job deadlines start from the Job controller's start time. The future controller must also enforce the plan's absolute three-hour deadline by canceling the attempt; the adapter refuses new ensures after that deadline. A single Job identity does not guarantee exactly-once application side effects. Preparation and publication must remain idempotent. See the official Job lifecycle documentation.

Controller integration contract

1. In PostgreSQL, acquire an execution slot and a unique lease generation; persist the attempt and immutable plan before any Kubernetes write.
2. Call EnsureWorkspace, supplying any previously recorded workspace identity. Persist its returned UID before calling Ensure for a phase. Each new adapter instance must reattach to the workspace this way first.
3. Persist returned phase identities and inspect them on subsequent reconciles. Lost leases cause reconciliation and reattachment, not a new attempt. Protect all controller transitions against stale generations. Calls mutating workspace attachment must be serialized; the adapter is not a lease implementation.
4. Persist cancellation before invoking Cancel, and prohibit further ensures after cancellation. Do not retry or clean up until pod shutdown is confirmed.
5. Persist sanitized logs, resolved commits, validated artifacts and the outcome before cleanup. Publication Job success alone must not expose artifacts.

Database attempts/slots/fencing, the controller process, preparation/publication commands and capability protocol, shared logs, artifact commit, orphan discovery, and deployment RBAC remain to be implemented. The existing Docker reaper must never manage Kubernetes attempts. No migration or production backend switch is included in this slice.

Use a dedicated workload namespace with tightly restricted mutation permissions. The plan annotation is an identity check, not a cryptographic trust boundary for arbitrary cluster writers. A NetworkPolicy-capable CNI is required. Kubernetes NetworkPolicies are additive: another policy selecting compiler pods can reopen traffic. Real network isolation tests must verify the complete namespace policy set before launch. Preparation and publication egress allowlists are still deployment work.

Verification

go test ./internal/execution/... covers reattachment, concurrent ensures, phase ordering, pod shutdown, cancellation, cleanup, resource loss, plan/UID conflicts, deadline preservation, invalid plans and compiler settings.

To validate objects against a real API without persisting resources:

GBB_TEST_KUBECONFIG=/path/to/kubeconfig \
GBB_TEST_KUBE_NAMESPACE=default \
go test ./internal/execution/kubernetes -run TestAPIServerDryRun -v

Server-side dry runs of all three Jobs, the PVC and NetworkPolicy passed against Kubernetes v1.34.6+k3s1. The complete Go suite passed; external-service tests still skip unless configured. These dry runs do not prove PVC provisioning, scheduling, network isolation, real compilation, log persistence or recovery under controller death. Those acceptance tests require the remaining controller and phase workers.