TL;DR

Backend — a single FastAPI app. Workers — a separate singleton container with eight loops. Frontend — three apps (landing on Astro, dash on React 19 + TanStack Router, admin on React 19) behind one Caddy with TLS, per-host CSP, and 1h WS timeouts for long pipelines. Postgres 16 + ChromaDB + Cloudflare R2 — main storage. 61 migrations, Plan-100 control plane with coordinator/runner/reconciler.

Stack and architecture

Topology

Caddy (TLS + per-host CSP + WS timeouts 1h, flush_interval -1)
  ↓
  ├── landing.goragen.com   → Astro static
  ├── app.goragen.com       → React 19 SPA (dash)
  ├── admin.goragen.com     → React 19 SPA (admin)
  └── api.goragen.com       → FastAPI + WS endpoints
                              ↓
                              ↓     workers (replicas:1, stop_grace 120s)
                              ↓     ├── coordinator_loop
                              ↓     ├── runner_loop
                              ↓     ├── reconciler_loop
                              ↓     ├── auto_provisioner_loop
                              ↓     ├── daily_cleanup_loop
                              ↓     ├── distribution_worker
                              ↓     ├── voice/lip_sync workers
                              ↓     └── heartbeat_loop
                              ↓
                              ├── Postgres 16 (psycopg v3)
                              ├── ChromaDB (vectors)
                              └── External:
                                    Vast.ai (GPU rental)
                                    Kling Direct API
                                    Anthropic Claude
                                    Google Gemini
                                    Cloudflare R2
                                    Sentry

API (FastAPI)

api/main.py — entry point. Startup sequence:

1. Run SQL migrations (61 files, idempotent, ordered, pg_advisory_lock(42) for multi-replica safety).
2. Migrate legacy vast_instance.json singleton config to the vast_instances table.
3. Initialise auth tables, seed admin from ENV, seed the __system__ sentinel user (for FK on users(id) in system records).
4. reconcile_interrupted_on_startup() — re-queue jobs left in running after an unclean shutdown.
5. Restore in-flight WS sessions from DB, mark hung ones as interrupted.
6. Generate voice samples on first boot (if none).
7. Start scheduler_loop (other loops live in the workers container).

On shutdown: cancel app.state.bg_tasks, broadcast WS close code 1012 ("Service Restart") to active sessions. The dash client sees 1012 and reconnects immediately — without a timeout wait.

Workers container

A separate sidecar, replicas: 1 (singleton). This matters — some loops hold state in memory (lease tracking, in-progress provisions) that can't be shared without complex coordination.

The host:

• coordinator_loop — admission control. Reads instance_requests, applies parallel limits + velocity guard, INSERTs provision_attempts. Leader-elected via provision_leader.
• runner_loop — execution. Pulls claimed attempts, calls scripts/vast_provision.ensure_instance(...), applies the terminal reducer (success → ready, failed → terminated, etc).
• reconciler_loop — recovery. Scans stale leases across five source kinds, force-terminates stuck cancelling instances, releases destroyed instances.
• auto_provisioner_loop — minute tick, rule-based spawn/destroy. Currently emits through the control plane on source_kind='auto_pool'.
• daily_cleanup_loop — 24h retention sweep: job_queue, decisions, lora_workspaces, idempotency keys, domain events >180d, disk usage check.
• distribution_worker — 30s tick, scheduled posts → social platforms.
• voice_clone_worker / lip_sync_worker — voice job mocks (for future full inference).
• heartbeat_loop — every 30 seconds writes system_settings.workers_last_heartbeat.

SIGTERM → 120s graceful. Coordinator, runner, and reconciler observe heartbeat_task.done() so leadership loss doesn't stay silent.

Database

PostgreSQL 16, access via api/db.py:get_connection — a psycopg v3 wrapper with SQLite-style API. This is needed because legacy runtime SQL was written for SQLite (row[0], row["col"], ? placeholders); api/db_dialect.translate_runtime_sql rewrites placeholders + datetime('now') on every execute.

This is a compromise: instead of rewriting 100k+ lines of SQL — a runtime translator. The Phase 11→12 cutover (2026-05-12) removed the dual driver.

Migrations

61 SQL files in api/migrations/:

• 001–027 — Phase 0–7 foundation: auth, workspaces, plans, personas, storyboards, gallery, distribution, voice, API keys, webhooks.
• 028–029 — Phase 8.1: job_queue, instance LoRA cache.
• 030–032 — Phase 8.2: vast_instances, provisioning_rules, provisioner_decisions.
• 033 — Phase 8.3: lora_training_jobs.
• 034–037 — Phase 9: domain_events, idempotency_keys, user_daily_spend, cost-protection knobs.
• 040–044 — Phase 11→12: Postgres-cutover artefacts.
• 047–054 — Phase 14 slim_provisioning: model_registry_v2, node_packages, presets, bootstrap meta, Telegram bot storage.
• 055–061 — Plan-100 Stage 1–6: shadow tables, atomic reuse claim, cutover flags, observability thresholds.

Each migration is idempotent, a _migrations tracker table, pg_advisory_lock(42) serialises multi-replica startups.

ChromaDB

knowledge/db/chroma/ — vector index for prompt recipes and Civitai data. 2,458 ComfyUI node schemas cached in node_info_cache.py. 40 reference workflows (WAN 2.1, HunyuanVideo, LTX 2.3) live in knowledge/db/workflow_examples/.

knowledge/workflow_builder.py — programmatic ComfyUI graph assembly; patch_workflow_with_lora injects LoRA loaders into a base workflow.

Authentication

Two modes:

1. Cookie JWT — primary, browser flow. access_token + refresh_token on .goragen.com. Argon2 for passwords.
2. Bearer gk_live_* — SDK / CI. Argon2-hashed keys in api_keys.key_hash. Falls through get_current_user after the cookie path.

require_admin, require_plan(...) — depends decorators for authorisation.

Middleware

• rate_limit.py — slowapi keyed by JWT user → API key prefix → IP.
• idempotency.py — Phase 9 C5 @idempotent(ttl_hours=24). Reads Idempotency-Key, looks up idempotency_keys BEFORE running the handler. Hit → replay cached JSONResponse; hash mismatch (same key, different body) → 409; missing header → 400. Applied to 5 mutating endpoints. The dash client auto-injects crypto.randomUUID() for these paths and re-uses the key on 401/429/5xx retries.
• maintenance_middleware — global 503 when system_settings.maintenance_mode == 'true' (admin/auth/health passthrough).
• CORS — pinned to ["GET","POST","PATCH","DELETE","OPTIONS"].
• _record_unhandled — feeds unhandled exceptions into the error_buffer ring buffer for the admin dashboard.

Frontend

apps/dash — user dashboard

React 19 + TypeScript + Vite 6 + Tailwind 4. Routing — TanStack Router (file-based, flat).

State split:

• TanStack Query — server cache, refetch, mutations.
• Zustand — cross-component session signals (live cost ticker).
• lib/_http.ts — HTTP client, handles 401/refresh, 429 retry, 5xx exponential backoff, offline detection.

WebSocket reconnect — backoff [1s, 2s, 5s, 10s, 30s] in src/lib/ws.ts.

Routes (high-level): /, /library, /library/$slug, /generate, /storyboard, /gallery, /voice, /lora, /distribution, /compliance, /settings/* (7 sub-routes).

Onboarding tour is lazy-loaded via Driver.js.

apps/admin — admin SPA

React 19. Sidebar shell AdminShell.tsx with keyboard shortcuts.

Routes: /dashboard, /users, /personas, /presets, /workflows, /instances, /tasks (4 sub-tabs), /lora, /system (6 sub-tabs including control-plane), /compliance, /settings.

apps/landing — marketing

Astro static. No SPA runtime. Talks only to /auth/register, /auth/login, /auth/forgot-password.

Caddy

Single TLS terminator. Per-host CSP. Reverse-proxy to four services. The critical setting for long pipelines:

read_timeout 1h
write_timeout 1h
flush_interval -1

LoRA training takes 45–90 minutes, storyboard — 10–15 minutes. Without these timeouts the WS session would drop every 30–60 seconds.

External

• Vast.ai — rented GPU pool. Provisioning via scripts/vast_provision.py. SSH ControlMaster + retry helpers in ssh_utils.py.
• Kling Direct API — endpoints for T2V, I2V, omni, extend, lip-sync, effects. JWT (HS256) in KlingClient.
• Anthropic Claude — orchestration (model selection, prompt engineering, captions).
• Google Gemini 2.5 Pro — output evaluation (4 criteria, verdict).
• Cloudflare R2 — canonical storage (STORAGE_BACKEND=s3, bucket goragen-prod, CDN at cdn.goragen.com).
• Sentry — error tracking (no-op when SENTRY_DSN is empty).

What I'd rewrite

— Workers as a single replica. Today it's a singleton by necessity (in-memory state). It could be broken into per-loop containers with split responsibility — each loop a separate process with persistent state in Postgres. But it's a significant rewrite and not a current blocker.

— Workflow templates as code. Today reference workflows are JSON files. A DSL could be built for assembling graphs from primitives (camera move + model + lora + sampler) that compiles to JSON.

— Self-hosted Kling alternative. Kling is a closed API. When an open-source alternative catches up (WAN 3.x / Hunyuan v2), the model_selector can be switched over without changes on top.

The platform runs in production at goragen.com. 61 migrations means six months of active development with regular zero-downtime deploys (the Phase 11→12 cutover was critical, completed in chunks).