Lab Skimmer

Self-hosted auction trawler for homelab/server hardware. It polls government surplus and datacenter-decommission auction sites, normalizes every listing into one schema, evaluates configurable rules against it, and surfaces the matches on a dashboard you read once a morning.

Single tenant, single operator. No accounts, no public exposure — it runs on a personal k3s cluster behind an internal ingress.

What it does

1. Collect — per-site adapters fetch listings (official JSON API first, HTML scrape as fallback), honoring robots.txt and a per-site rate limit.
2. Normalize — listings land in one listings table keyed by (source, external_id); prices are integer cents, unknown values stay NULL.
3. Dedupe — the same auction seen on two sources is linked to one canonical row by normalized title or URL path.
4. Evaluate — every enabled rule (keyword / price threshold / price-per-spec score) runs against each listing; hits become matches rows.
5. Rank by reach — each listing's pickup location is resolved to miles from home at ingest, and the deals endpoint scales every score by how far away it is, so a great deal you would have to cross the country for ranks below a good one down the road. Tunable without a re-sweep.
6. Surface — a SvelteKit SPA reads the Go API and shows the morning scan, ranked by the distance-adjusted score and sortable by any column, with per-source collector health.
7. Track — anything worth watching goes on a watchlist, whether the sweep found it or you pasted the URL in from a site the registry has never heard of; every sweep re-reads those pages so prices stay current.

Rules are data, not code. Change them through the API or the dashboard's Rules page; no redeploy. Both rules and sources are seeded from config/*.seed.json on every boot and editable at runtime — see docs/configuration.md.

Architecture at a glance

 auction sites                 k3s: namespace labskimmer
 ─────────────                 ────────────────────────────────────────────
 publicsurplus.com ┌────────────────────────┐        ┌──────────────────┐
 municibid.com     │ CronJob collector-sweep│        │ Deployment       │
 hibid.com  ▲      │ /server --sweep-once   │        │ backend          │
 api.gsa.gov│HTTPS │  06:00 daily           │        │  Gin API :8080   │
 (any site) │      │  collect → normalize   │        │  + static SPA    │
        └──────────┤  → dedupe → rules      │        │  + Run-now sweep │
                   │  → watchlist refresh   │◄───────┤    (same code)   │
                   └───────────┬────────────┘        └────────┬─────────┘
                               │        both use pgx          │
                               ▼                              ▼
                        ┌──────────────────────────────────────────┐
                        │ PostgreSQL 16 (shared cluster instance)   │
                        │ sources listings bid_snapshots            │
                        │ rules matches source_health               │
                        │ favorites sweep_runs                      │
                        └──────────────────────────────────────────┘
                                            ▲
                                   Ingress (Traefik) ── front proxy (TLS)
                                   https://lab-skimmer.picklemustard.dev

One binary, two modes: the long-running API server, and --sweep-once for the CronJob. Both apply goose migrations at boot, so there is no separate migrate job. The dashboard's Run button sweeps in the API pod using the same SweepRunner; the CronJob's schedule and behavior are untouched by it.

Details: docs/architecture.md.

Repo layout

Path	What lives here
cmd/server	entrypoint; API mode and --sweep-once mode
internal/api	Gin handlers, static SPA serving with SPA fallback
internal/collect	site adapters, rate limiter, robots.txt, retry, sweep, dedupe
internal/rules	rule engine: keyword, threshold, spec-score; seeding
internal/geo	embedded US gazetteer; free-text location to miles from home
internal/store	sqlc-generated queries and models
internal/config	defaults < JSON file < LABSKIM_* env precedence
db/migrations	goose migrations (the schema source of truth)
db/queries	sqlc input SQL
web	SvelteKit SPA (adapter-static)
deploy/base, deploy/overlays/cluster	Kustomize manifests
.forgejo/workflows	CI: test, image publish, release
scripts	smoke/E2E/manifest validation, CI annotation helpers

Quick start

Whole stack in containers (Postgres + backend, migrations on boot):

make compose-up          # docker or podman, whichever is installed
curl localhost:8080/healthz
make compose-down

Backend against a local Postgres, dashboard in dev mode:

make migrate-up                                   # goose, needs DATABASE_URL
go run ./cmd/server                               # API on :8080
cd web && npm install && npm run dev              # SPA on :5173, /api proxied

One sweep by hand — from a shell, or with the dashboard's Run now button:

go run ./cmd/server --sweep-once

More: docs/development.md.

Configuration

Precedence: built-in defaults < JSON file (LABSKIM_CONFIG_FILE) < LABSKIM_* env vars. The usual knobs:

Env var	Default	Meaning
LABSKIM_DATABASE_URL	postgres://labskimmer:labskimmer@localhost:5432/labskimmer?sslmode=disable	Postgres DSN
LABSKIM_HTTP_ADDR	:8080	API listen address
LABSKIM_DASHBOARD_DIR	empty (API-only)	static SPA directory; /dashboard in the image
LABSKIM_MIGRATIONS_DIR	db/migrations	goose migrations, /db/migrations in the image
LABSKIM_USER_AGENT	lab-skimmer/0.1 (…)	honest UA sent to auction sites

Full table plus rule config shapes: docs/configuration.md.

API

Same origin as the dashboard, everything under /api/v1.

Method	Path	Purpose
GET	/healthz	liveness/readiness, 200 ok; ?verbose=1 adds Vault seal state
GET	/api/v1/deals	matched deals, filterable, ranked by distance-adjusted score
GET	/api/v1/listings/:id	listing detail, matches, latest bid snapshot
GET	/api/v1/rules	list rules
POST	/api/v1/rules	create a rule — auth
PUT	/api/v1/rules/:id	replace a rule — auth
DELETE	/api/v1/rules/:id	delete a rule — auth
GET	/api/v1/favorites	the watchlist
POST	/api/v1/favorites	track an auction, by listing or by URL — auth
PATCH	/api/v1/favorites/:id	edit notes / target price — auth
DELETE	/api/v1/favorites/:id	stop tracking — auth
POST	/api/v1/favorites/:id/refresh	re-read one item's page now — auth
GET	/api/v1/sweeps	recent sweep runs
GET	/api/v1/sweeps/latest	most recent sweep run
POST	/api/v1/sweeps	run a sweep now — auth
GET	/api/v1/sources/health	per-source collector status
GET	/api/v1/sources	list the source registry
POST	/api/v1/sources	create a source
PUT	/api/v1/sources/:id	replace a source
DELETE	/api/v1/sources/:id	delete a source
POST	/api/v1/sources/:id/test	dry-run a source against the live site

Reads are open; every write needs Authorization: Bearer $LABSKIM_API_TOKEN. A source is an operator-supplied URL the server fetches on a schedule, so with no token configured the write endpoints return 503 rather than running open.

Request/response shapes: docs/api.md.

Watchlist

Anything worth keeping an eye on goes on the Watchlist page: star it on the morning scan, or paste the URL of an auction spotted on a site the registry has never heard of. Both kinds live in one list with notes and an optional target price, and both are re-read on every sweep — the daily one and any run started by hand.

A pasted URL is read the way a crawler would: JSON-LD schema.org/Product first, then OpenGraph and product meta tags, then <title>. What the page does not state stays unknown rather than being guessed, and a page that cannot be read at all is still tracked as a link — with the reason shown on the row — because a site that publishes nothing for crawlers is exactly the case worth keeping in one place.

Refreshing honors the same politeness a sweep does: robots.txt per URL, and per-host request spacing taken from the source's min_request_interval_ms.

Manual runs

Run now on the morning scan starts a sweep in the API pod and reports what it did — listings upserted, matches, watchlist items checked, and any source that failed by name. It returns immediately and the dashboard polls, because a full sweep is minutes of deliberately rate-limited requests.

The 06:00 CronJob is a separate pod on its own schedule and is not affected: pressing this neither delays, replaces, nor skips it. Both paths now record a row in sweep_runs, which is how the dashboard can answer "when did this last update?" across pod restarts.

Per-source API keys

A site that needs a credential gets a ${vault:path#key} reference in its config, resolved against HashiCorp Vault at request time:

"headers": { "Authorization": "Bearer ${vault:lab-skimmer/govdeals#api_key}" }

The config holds only the reference, so it stays safe to commit and safe to render in the dashboard; the resolved value is masked in every error and in the dry-run panel. Vault being sealed or unconfigured fails that source alone — sources needing no credential are untouched.

On the cluster the app authenticates to Vault with its Kubernetes service account, so no Vault credential is stored in the cluster at all; compose and local runs use AppRole.

Reference syntax, failure modes, and Vault setup: docs/configuration.md and docs/runbook.md.

Networking

Ingress host lab-skimmer.picklemustard.dev (wildcard DNS → front proxy terminates TLS → Traefik → backend:8080). Outbound: the two auction sites plus their robots.txt, minimum 2 s between requests per site. Database and container-registry traffic stay in-cluster.

Every port, hostname, and the CI runner's registry-reachability quirk: docs/networking.md.

Deploy and operate

Images are published to the Forgejo registry by CI; the cluster overlay pins an explicit tag and bumping it is the deploy. Nothing in CI touches the cluster.

make cluster-deploy      # kubectl apply -k deploy/overlays/cluster
./scripts/e2e-cluster.sh # healthz, manual sweep, source health, deals, dashboard

First-time setup, secrets, image updates, CI failure triage, troubleshooting: docs/runbook.md.

Testing

make test            # go test ./...
make cover-ci        # what CI runs: -race, coverage summary
make kustomize-validate
make smoke           # compose stack must answer /healthz
cd web && npm test   # vitest; npm run test:e2e for Playwright

Store tests boot an embedded PostgreSQL on port 15433 and apply the real migrations — no external database needed.

Documentation

Doc	Covers
docs/architecture.md	components, data flow, schema, rule engine, dedupe
docs/networking.md	ports, ingress, DNS, egress, registry, CI network
docs/api.md	HTTP endpoints, parameters, payloads, status codes
docs/configuration.md	config precedence, env vars, rule configs, seeding
docs/development.md	local setup, make targets, codegen, tests
docs/runbook.md	k3s deploy, operations, CI, troubleshooting