86 lines
5.9 KiB
Markdown
86 lines
5.9 KiB
Markdown
# Events go through a transactional outbox and are delivered idempotently
|
|
|
|
Domain events used to be published after the write that caused them, from the use case, on a
|
|
channel the server owned. A crash between the write and the publish lost the event (no poster, no
|
|
federation delivery, no search index entry), and with NATS the server needed a live broker
|
|
connection just to log a review. Delivery was also at-least-once without any guard, so a redelivered
|
|
event ran its handlers twice.
|
|
|
|
## Decision
|
|
|
|
**The outbox is `event_queue`.** The same SQLite table the DB-backed bus already used is now the
|
|
outbox for every backend. Migration `0101` adds `event_queue.event_id` (UUID, unique when set) and
|
|
`claimed_at`, and creates `processed_events` (`event_id`, `handler`, `status`, `started_at`, owner
|
|
token). Legacy rows have a NULL `event_id` and are still delivered; they just get no dedup.
|
|
|
|
**Writes are atomic with their events.** Repository and command-port methods that change state take
|
|
the events they cause and write them in the same transaction as the data. `shared::enqueue_events`
|
|
is the single path that inserts outbox rows. Covered: review log/edit/delete (including orphan
|
|
movie deletion and `ReviewDeleted`/`MovieDeleted`), watchlist add/remove, movie discovery and
|
|
enrichment requests, wrap-up, integration writes, local follow/unfollow, remote review save.
|
|
Use cases never open transactions; the adapter does.
|
|
|
|
**The server never talks to NATS.** With `EVENT_BUS_BACKEND=nats` the server still only inserts
|
|
outbox rows. The worker runs a relay that claims pending rows and publishes them to JetStream with
|
|
`Nats-Msg-Id` set to the event id, then marks the row done. With `db` the worker consumes the rows
|
|
directly. `max_deliver` on the JetStream consumer equals `EVENT_QUEUE_MAX_ATTEMPTS`, so both
|
|
backends dead-letter after the same number of attempts.
|
|
|
|
**Three dedup layers.**
|
|
1. *Broker.* JetStream drops a repeated `Nats-Msg-Id` inside `duplicate_window`, set explicitly to 1
|
|
hour (longer than the relay's retry horizon) rather than the 2-minute server default.
|
|
2. *Relay lease.* A row claimed by a relay or DB consumer that then died is reclaimed once
|
|
`claimed_at` is older than 600 s, so a crash never strands an event in `processing`.
|
|
3. *Processed-event ledger.* Before a handler runs, `(event_id, handler)` is claimed in
|
|
`processed_events` (INSERT `in_progress`, or take over a claim older than the 1800 s lease).
|
|
Success marks it `done`; failure releases it; both are conditioned on an owner token so a
|
|
stale worker cannot clobber a newer claim. A delivery that sees a fresh claim held elsewhere is
|
|
deferred (`AckHandle::defer`: rescheduled to the lease expiry without consuming an attempt;
|
|
JetStream `nak` with delay, which does count toward `max_deliver`). A `done` row skips the handler.
|
|
Ledger rows are kept 30 days. A ledger read error skips the handler and nacks.
|
|
|
|
## Guarantees
|
|
|
|
- No two deliveries run the same handler for the same event concurrently.
|
|
- At-least-once after a crash: a lost claim expires and the event is retried.
|
|
- Handlers are idempotent, so the remaining duplicate window (crash after the effect, before the
|
|
ledger mark) is harmless. The O5 audit made each handler safe to re-run: image cleanup (NotFound
|
|
is Ok), image conversion (probes the exact key; missing original means done), follow backfill and
|
|
inbound review import (upserts, deterministic Create ids), person/movie enrichment (upserts; the
|
|
movie profile write is last because it is the "done" marker), wrap-up (a `Generating` record is
|
|
resumed; completion is atomic so `WrapUpCompleted` is emitted once), search handlers (keyed
|
|
upserts and removes), poster sync (skips when the poster is already stored under the movie id, so
|
|
no second `PosterSynced`/`ImageStored`).
|
|
|
|
## Known gaps
|
|
|
|
- **Remote and block follow events are best-effort.** `CompositeSocialAdapter` routes remote
|
|
follows and blocks through k-ap, which owns that transaction. Its events are published after the
|
|
fact through an injected publisher, so a crash between the k-ap write and the publish loses them.
|
|
Fixing it needs transaction hooks in k-ap.
|
|
- **k-ap 0.5.0 mints random ids for some activities.** Create, Announce and Like ids are
|
|
deterministic (`broadcast.rs:22-35`, `:283-288`; backfill `backfill.rs:208-212`). Delete
|
|
(`broadcast.rs:157`), Add/Undo (`:210`), Update (`:321`), Move (`:377`) and the actor update
|
|
(`:339-344`) use `Uuid::new_v4()` via `url_scheme.rs:52-53` with no override; only
|
|
`broadcast_raw_to_followers` (`:395`) takes a caller id. A duplicate of those after a crash is a
|
|
fresh-id activity. They are semantically idempotent for remote servers (deleting twice, updating
|
|
to the same state), but it is a new activity, not a replay. The ledger makes this rare; closing it
|
|
means building those activities ourselves or a k-ap change.
|
|
- **Core NATS has no safety net.** `NATS_MODE=core` is fire-and-forget: no broker dedup, no
|
|
redelivery. Rows are marked done after publish; a subscriber that is down misses the event. The
|
|
ledger still prevents double handling of a repeated id.
|
|
- **`ImageStored` stays best-effort.** It is raised after an object-storage write, which no
|
|
database transaction can cover.
|
|
|
|
## Consequences
|
|
|
|
- Existing deployments keep their JetStream stream. `get_or_create_stream` applies config only on
|
|
creation, so the consumer now compares an existing stream and consumer with what this build needs
|
|
and updates `duplicate_window` (1 h) and the consumer's `max_deliver`/`ack_wait` in place, with a
|
|
warning in the log. Both are mutable on a live stream/consumer, so no recreate is needed;
|
|
immutable consumer fields (name, filter, deliver policy) are never touched.
|
|
- Starting server and worker at the same moment against a database that still needs migration `0101`
|
|
can make one fail with `duplicate column name`; start the server first, or restart the loser.
|
|
- Handler authors must keep new handlers idempotent; the ledger narrows duplicates but cannot
|
|
remove the crash-after-effect case.
|