# Events go through a transactional outbox and are delivered idempotently

Domain events used to be published after the write that caused them, from the use case, on a
channel the server owned. A crash between the write and the publish lost the event (no poster, no
federation delivery, no search index entry), and with NATS the server needed a live broker
connection just to log a review. Delivery was also at-least-once without any guard, so a redelivered
event ran its handlers twice.

## Decision

**The outbox is `event_queue`.** The same SQLite table the DB-backed bus already used is now the
outbox for every backend. Migration `0101` adds `event_queue.event_id` (UUID, unique when set) and
`claimed_at`, and creates `processed_events` (`event_id`, `handler`, `status`, `started_at`,
`processed_at`). The migration backfills every existing row with a random event id, so legacy rows get dedup too.
The column stays nullable, but the only insert path (`enqueue_events`) always sets it, and the
consumer still tolerates a NULL id (no dedup) for rows from foreign writers.

**Writes are atomic with their events.** Repository and command-port methods that change state take
the events they cause and write them in the same transaction as the data. `shared::enqueue_events`
is the single path that inserts outbox rows. Covered: review log/edit/delete (including orphan
movie deletion and `ReviewDeleted`/`MovieDeleted`), watchlist add/remove, movie discovery and
enrichment requests, wrap-up, integration writes, local follow/unfollow, remote review save.
Use cases never open transactions; the adapter does.

**The server never talks to NATS.** With `EVENT_BUS_BACKEND=nats` the server still only inserts
outbox rows. The worker runs a relay that claims pending rows and publishes them to JetStream with
`Nats-Msg-Id` set to the event id, then marks the row done. With `db` the worker consumes the rows
directly. `max_deliver` on the JetStream consumer equals `EVENT_QUEUE_MAX_ATTEMPTS`, and a JetStream
nak uses the same backoff as the SQLite queue (`domain::events::retry_backoff`: 5 s doubling per
attempt, capped at 300 s), so both backends give up after the same number of deliveries on the same
schedule. They differ in what giving up means. SQLite marks the row `dead_lettered`. JetStream has no
DLQ: after `max_deliver` the message is no longer delivered but stays in the stream (until stream
limits evict it). The consumer logs `event exhausted max deliveries` at error level with the event id
on the final nak. To find such messages, watch the JetStream max-deliveries advisory
(`$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.<stream>.<consumer>`) or inspect the stream
(`nats stream view <stream>`) for sequences the consumer has not acked. The relay never spends
attempts on transport failures: when the bus is unreachable it defers the row with growing backoff and
pauses, so an outage cannot dead-letter rows.

**Three dedup layers.**
1. *Broker.* JetStream drops a repeated `Nats-Msg-Id` inside `duplicate_window`, set explicitly to 1
   hour (longer than the relay's retry horizon) rather than the 2-minute server default.
2. *Relay lease.* A row claimed by a relay or DB consumer that then died is reclaimed once
   `claimed_at` is older than 600 s, so a crash never strands an event in `processing`.
3. *Processed-event ledger.* Before a handler runs, `(event_id, handler)` is claimed in
   `processed_events` (INSERT `in_progress`, or take over a claim older than the 1800 s lease).
   Success marks it `done`; failure releases it; both are conditioned on the claim's millisecond
   `started_at` value (`WHERE ... AND started_at = ?`), which acts as the owner token, so a stale
   worker cannot clobber a newer claim. A delivery that sees a fresh claim held elsewhere is
   deferred (`AckHandle::defer`: rescheduled to the lease expiry without consuming an attempt;
   JetStream `nak` with delay, which does count toward `max_deliver`). A `done` row skips the handler.
   Ledger rows are kept 30 days. A ledger read error skips the handler and nacks.

## Guarantees

- No two deliveries run the same handler for the same event concurrently.
- At-least-once after a crash: a lost claim expires and the event is retried.
- Handlers are idempotent, so the remaining duplicate window (crash after the effect, before the
  ledger mark) is harmless. The O5 audit made each handler safe to re-run: image cleanup (NotFound
  is Ok), image conversion (probes the exact key; missing original means done), follow backfill and
  inbound review import (upserts, deterministic Create ids), person/movie enrichment (upserts; the
  movie profile write is last because it is the "done" marker), wrap-up (a `Generating` record is
  resumed; completion is atomic so `WrapUpCompleted` is emitted once), search handlers (keyed
  upserts and removes), poster sync (skips when the poster is already stored under the movie id, so
  no second `PosterSynced`/`ImageStored`).

## Known gaps

- **Remote and block follow events are best-effort.** `CompositeSocialAdapter` routes remote
  follows and blocks through k-ap, which owns that transaction. Its events are published after the
  fact through an injected publisher, so a crash between the k-ap write and the publish loses them.
  Fixing it needs transaction hooks in k-ap.
- **k-ap 0.5.0 mints random ids for some activities.** Create, Announce and Like ids are
  deterministic (`broadcast.rs:22-35`, `:283-288`; backfill `backfill.rs:208-212`). Delete
  (`broadcast.rs:157`), Add/Undo (`:210`), Update (`:321`), Move (`:377`), the Undo wrappers for
  Announce (`:81`) and Like (`:129`) and the actor update (`:339-344`) use `Uuid::new_v4()` via `url_scheme.rs:52-53` with no override; only
  `broadcast_raw_to_followers` (`:395`) takes a caller id. A duplicate of those after a crash is a
  fresh-id activity. They are semantically idempotent for remote servers (deleting twice, updating
  to the same state), but it is a new activity, not a replay. The ledger makes this rare; closing it
  means building those activities ourselves or a k-ap change.
- **Core NATS has no safety net.** `NATS_MODE=core` is fire-and-forget: no broker dedup, no
  redelivery. Rows are marked done after publish; a subscriber that is down misses the event. The
  ledger still prevents double handling of a repeated id.
- **`ImageStored` stays best-effort.** It is raised after an object-storage write, which no
  database transaction can cover.

## Consequences

- Existing deployments keep their JetStream stream. `get_or_create_stream` applies config only on
  creation, so the consumer now compares an existing stream and consumer with what this build needs
  and updates `duplicate_window` (1 h) and the consumer's `max_deliver`/`ack_wait` in place, with a
  warning in the log. Both are mutable on a live stream/consumer, so no recreate is needed. Each update is a
  clone of the existing config with only those fields changed, so operator-set fields and the
  immutable ones (name, filter, deliver policy) are preserved. If an update is rejected the worker
  logs a warning with the manual `nats stream edit` / `nats consumer edit` command and keeps
  starting.
- Server and worker both migrate at startup (docker compose starts them together). `sqlite::migrate`
  retries with a bounded backoff (100 ms to 1 s, 5 retries), so the loser of a migration race
  re-reads `_sqlx_migrations`, skips what the winner applied and succeeds; only a persistent error
  is returned.
- Handler authors must keep new handlers idempotent; the ledger narrows duplicates but cannot
  remove the crash-after-effect case.