1 Reference Durable Execution Store
picklemustard edited this page 2026-09-07 15:49:27 -04:00

Durable execution persistence

Migration 0007_execution_attempts.sql adds Kubernetes attempt records and two execution slots without rewriting historical jobs or specs. This is a controller persistence foundation; the production queue still uses Docker.

StartExecution records a validated, immutable plan and deadline before any Kubernetes mutation. Concurrent callers reserve at most two slots. An identical start is idempotent; changed plans conflict. Attempts are limited to two, each with a database-constrained deadline of at most three hours.

LeaseExecution acquires an unfinished attempt using a new database sequence generation. Lease expiry preserves its slot, phase, plan, and Kubernetes UIDs. It returns canceled and overdue attempts too, so the controller can stop their pods. Resource, phase, progress-log, and outcome writes reject expired or superseded generations. A persisted resource UID cannot be replaced.

The controller must inspect Kubernetes and confirm that preceding pods stopped before advancing phases. Compilation additionally requires persisted engine/module commit pins; phase callbacks record them through a generation-fenced write. It must confirm all pods stopped and logs persisted before FinishExecution. That transaction serializes cancellation with artifact publication and releases the execution slot. Successful publication sets the configured artifact retention deadline. Failed attempts may retry once when the admission-locked waiting queue has room; cancellation never retries. The legacy Docker claimant skips enrolled jobs.

Completed attempts remain in a cleanup query until MarkExecutionCleaned follows successful UID-checked workload deletion and partial-artifact cleanup. Recorded plans and UIDs remain available for administrative history.

The real PostgreSQL tests exercise competing starts, lease takeover, expired writes, UID conflicts, sanitized-log replay, retry limits, cancellation during publication, and cleanup records. They do not establish Kubernetes shutdown, compiler isolation, object-store publication, or controller restart acceptance.

Still required: the controller process and deployment, phase source-report integration, phase workers and scoped storage capabilities, actual log collection, publication validation/retention, orphan reconciliation, and cluster tests.