Infrastructure

What runs, why it runs, and what it would take to remove it. Local is Docker Compose; production is Kubernetes with the same images.

The pieces

The MinIO image (#308)

No public registry serves the MinIO image prod runs (quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z): Docker Hub answers 401 and quay.io refuses anonymous pulls. The prod droplet's cached copy was copied, layer for layer, by .github/workflows/mirror-minio.yml to:

ghcr.io/verjson/marketing-studio/minio@sha256:a1a8bd4ac40ad7881a245bab97323e18f971e4d4cba2c2007ec1bedd21cbaba2

The package is private and owned by this repository, like marketing-studio/api, so a droplet logged in to GHCR can pull it. docker-compose.prod.yml reads MINIO_IMAGE. A droplet with no cached copy (nonprod, a rebuilt prod) sets it to the reference above. Unset, it falls back to prod's existing image, so prod's MinIO is never recreated by this change.

The Google Cloud copy of each candidate (#381)

The org's release workflow will only promote a candidate that also exists in Google Artifact Registry (container-release.yml refuses a candidate without exactly one GAR destination). So every candidate is also copied, digest for digest, to:

us-central1-docker.pkg.dev/verjson-ci-640463/marketing-studio-candidates/marketing-studio-candidate-{api,web}

It lives in verjson-ci's Google Cloud project, next to verjson-ci's own container-canary repository, and is set up the same narrow way:

The trust condition names the contract SHA. When the generated files move to a new Verjson/.github pin, the publish fails at Google sign-in until the condition names the new SHA too. Update it in the same change, with gcloud iam workload-identity-pools providers update-oidc marketing-studio-main --workload-identity-pool=github-oci-candidates --location=global --project=verjson-ci-640463 --attribute-condition=... (swap the SHA in the current condition, read with describe).

Two kinds of "background work", and why both exist

They are not competing choices — they solve different problems, and using one for the other's job goes badly in a predictable way.

RabbitMQ because nearly every consumer here talks to a rate-limited third-party API. Per-queue prefetch, fair dispatch and a native dead-letter path are exactly the primitives that need — all of which a polling table has to reimplement, usually badly. See D-014.

trigger.dev for the agentic layer. An agent run is long, multi-step, and has to survive a deploy and a wait-for-human — which is precisely what a durable workflow engine is for and what a queue is not. See D-016.

Postgres keeps the rate limiter (a fixed-window counter, not a queue) and every read-model.

Queue topology

verjson.publish        → publish a post to one channel      (per-channel queues, prefetch 1)
verjson.metrics        → pull metrics for one account/day
verjson.tokens         → refresh OAuth tokens expiring soon
verjson.scrape         → fetch and parse one URL            (politeness delay per host)
verjson.whatsapp       → send one approved template
verjson.dlq            → anything that exhausted its retries

Each queue is bound to a direct exchange by routing key. Retries use a delayed retry queue rather than an in-process sleep, so a redeploy does not lose the backoff.

WhatsApp: Baileys

Baileys speaks the WhatsApp Web protocol from a persistent socket, so it runs as its own container with its own volume — a long-lived paired session cannot share a lifecycle with stateless API pods that get rescheduled. The volume holds the pairing; losing it means re-pairing by QR code.

The honest trade-off. Baileys is an unofficial, reverse-engineered client. Bulk marketing over it is against WhatsApp's terms, and the realistic downside is the number being banned — not a rate-limit, a ban. It is the pragmatic choice for an internal tool and for conversational replies; it is the wrong choice for a large cold broadcast.

The mitigation is structural, not procedural: the bridge sits behind the same ChannelProvider interface as every other channel, so the official WhatsApp Business Cloud API is a transport swap rather than a rewrite. If volume grows or a ban lands, changing one adapter moves the whole feature across. See D-015.

Operationally: opt-in is enforced before anything is queued, sends are paced, and template content still passes the approval gate.

The bridge's surface

services/whatsapp — a standalone ESM Node 22 service, outside the npm workspace so it can hold a stale Baileys pin without dragging the API's dependency tree along with it. Baileys is pinned to 6.7.24: npm's latest tag is a 7.0.0 release candidate, and legacy is the last stable line.

Every route but /health requires x-bridge-token, compared in constant time. /health is open because it is the container's HEALTHCHECK and the StatefulSet's probe, and it reveals only that the process is up.

Outbound, the bridge POSTs to WEBHOOK_URL (default http://api:4000/api/v1/whatsapp/events) with the same x-bridge-token. Payloads are { event, occurred_at, data } and snake_case, because that is what the API validates inbound; the bridge's own REST responses follow the table above instead.

Delivery is fire-and-forget. A webhook failure is logged and dropped: it must never take the socket down or fail a caller's send, because the API is the bridge's audience, not its dependency. Durability for anything that matters belongs on the API side of that hop.

A dropped socket reconnects with exponential backoff to 30 s. A close carrying loggedOut (the number was un-linked from the phone) clears the auth state instead of looping on dead credentials, so the bridge lands back at "show a QR".

Scrapers

Scraping is a queue consumer like any other (verjson.scrape), which means it inherits retries, dead-lettering and rate limiting for free rather than growing its own scheduler.

Rules that keep it defensible:

• Respect robots.txt and a per-host politeness delay. Enforced in the consumer, not left to each scraper.
• Identify honestly in the user agent, with a contact URL.
• Cache aggressively. A page fetched today is not fetched again tomorrow unless something asks for a refresh.
• Public pages only. Nothing behind a login, and nothing a robots.txt disallows.
• Personal data has a retention clock from the moment it lands — see the Apollo note in FEATURES.md (F-092).

Agent-Reach

Agent-Reach is a capability layer that gives agents unified read access across ~14 platforms (X, Reddit, LinkedIn, YouTube, GitHub, RSS, podcasts, web search) by routing to native CLI tools with ordered fallbacks, rather than wrapping them in a new abstraction.

Where it fits here: the research half of the growth-intelligence work (F-091, F-092, F-094) and the agentic layer (F-071). It is installable as an MCP server, which means our agents can reach it through the same MCP path they already use — no bespoke integration, and no scraper of our own for platforms it already covers.

Where it does not fit: anything that writes. It is a read and search layer; publishing stays with our own channel adapters, behind the approval gate.

Tracked as F-098.

Nonprod (#379, #383)

A second droplet, ms-nonprod (144.126.249.116, Pulumi stack nonprod), runs the same Compose stack at https://nonprod.omnichan.studio (a DNS-only Cloudflare A record to the droplet; Caddy gets the certificate from Let's Encrypt). It is deployed only by dispatching deploy nonprod (.github/workflows/deploy-nonprod.yml), which builds nothing: it rolls an existing candidate's signed digests through prod's own deploy/roll-vm.sh. Its ref is a release tag (v0.9.0: the digests and commit come from that release's signed manifest, and /api/v1/version reports the release), a main commit, or empty for the latest candidate. Its GitHub environment nonprod holds its own SSH key and freshly generated secrets; none of prod's are shared. The database started empty; SEO providers run in mock mode. Prod's Caddyfile listens on :80 only, because its Cloudflare tunnel terminates TLS. Nonprod has no tunnel, so the deploy renders a copy whose site is nonprod.omnichan.studio, http://127.0.0.1 (the second address is what roll-vm.sh's health gate calls) and stages that; the routes come from the one repo file. The same happens for any environment with PUBLIC_SCHEME=https, a PUBLIC_HOSTNAME and no CLOUDFLARED_TOKEN. With no tunnel token, the cloudflared container restarts in a loop, which the health gate ignores.

Cutting a release (#382)

A release names one signed candidate that main already built; nothing is rebuilt. Dispatch container release (.github/workflows/container-release.yml, generated from Verjson/.github) from main with:

• version: X.Y.Z with no v. It must equal nextStableVersion in container-candidate.json at the candidate's commit, and be higher than the last release. After a release, next-version.yml moves nextStableVersion one patch on, straight to main with the release App (D-066); a minor or major release still needs a PR that sets the number. The next candidate carries it.

• candidate-manifest: <artifact id>@<artifact digest> of the container-candidate-<version>-rc.<run>.<attempt> artifact from that candidate's immutable container candidate run:

  gh api repos/verJSON/marketing-studio/actions/runs/<run>/artifacts \
    --jq '.artifacts[] | select(.name | startswith("container-candidate-")) | "\(.id)@\(.digest)"'
  

The run verifies the candidate (signature, GAR copy, expiry), retags both images as ghcr.io/verjson/marketing-studio-candidate-{api,web}:X.Y.Z, signs RELEASES/containers/vX.Y.Z.json, consumes every NEXT/ note into CHANGELOG/vX.Y.Z.md, and pushes that commit and the vX.Y.Z tag to main as the release-authorization App, then publishes the GitHub Release. Its key (RELEASE_APP_PRIVATE_KEY) lives only in the release-app environment, which only main can use. The release commit touches no app or deploy path, so prod does not redeploy. A failed NEXT/ note fails the run after the image tags are pushed; fix the note and re-run.

Deploying a release to prod (#384)

Dispatch deploy prod (.github/workflows/deploy-prod.yml) from main with release: vX.Y.Z. In order:

1. authorize: the dispatcher is in PROD_DEPLOYERS (in the workflow file; widening it is a reviewed change).
2. resolve: the release manifest carries the release workflow's signature; both images carry the publisher's signature for the candidate commit; the release has a successful deploy:release deployment on nonprod.
3. gates, against what /api/v1/version on prod reports: a release older than prod needs allow_downgrade (the run lists what it would take out); migrations prod has not run need migrations_acknowledged (the run lists them).
4. backup: db-backup.yml with kind: pre-deploy, to R2.
5. deploy: the prod .env (every prod setting, plus SOURCE_COMMIT and RELEASE_VERSION), then roll-vm.sh with the release's digests. MinIO keeps its cached image.
6. report: prod's /api/v1/version must serve the release's commit and tag; the run records a deploy:release deployment on prod.

This is the only way prod changes: merging to main deploys nothing (D-064). To roll back, dispatch it with the previous release and tick allow_downgrade; that restores the images, not the database.

The promoter (#385)

.github/workflows/promote.yml runs after every successful release and every six hours (:37). It skips while a deploy is running, holds a hop whose last deploy failed, then asks scripts/promote-plan.mjs what to do and writes the answer, with its reason, to the run summary.

• release → nonprod: a release tag newer than what nonprod runs is deployed there with deploy nonprod, by itself. PROMOTE_NONPROD=false turns this off. A candidate deployed to nonprod by hand holds both hops, so it is not replaced under someone looking at it.
• nonprod → prod: ready only when prod's commit is an ancestor of the release, nonprod has run it for PROMOTE_SOAK_HOURS (default 24), nonprod serves it, and it adds no migration. A migration opens one promotion-hold issue with the command to deploy by hand.

The prod hop is report-only ("ready for prod; automatic promotion is off") until both exist:

1. a fine-grained PAT owned by someone on PROD_DEPLOYERS, with actions: write on this repository only, as the repository secret PROMOTER_TOKEN (deploy-prod.yml refuses the bot account a GITHUB_TOKEN dispatch arrives as);
2. the repository variable AUTO_PROMOTE_PROD=true.

Check what it would do at any time: Actions → promote → Run workflow with dry_run, or node scripts/promote-plan.mjs --repo . --repository verJSON/marketing-studio locally.

Environment

Every value is listed in .env.example. The rule stands: server-side values are read only in apps/api/src/core/config.ts, and anything prefixed NEXT_PUBLIC_ is public and must never hold a secret.