perf: stream diary export instead of loading all entries into memory #5

New Issue

GKaszewski · 2026-06-10T09:21:40Z

GKaszewski commented

2026-06-10 09:21:40 +00:00

Problem

diary/export_diary.rs calls get_user_history() which loads the entire diary into a Vec<DiaryEntry> before serialization. For users with thousands of entries this holds everything in RAM at once.

Current flow:

DiaryRepository::get_user_history() → Vec<DiaryEntry> (all in memory)
DiaryExporter::serialize_entries(&[DiaryEntry], format) → Vec<u8> (serialized bytes, also in memory)
Response written from the Vec<u8>

Proposed fix

Change the DiaryExporter port to accept a stream/writer instead of a slice, and have the repository yield rows lazily:

DiaryRepository::stream_user_history() → BoxStream<DiaryEntry>
DiaryExporter::serialize_entries_stream(stream, format, writer) — writes directly to an AsyncWrite or returns a BoxStream<Bytes>
The HTTP handler pipes the stream directly into the response body (Axum supports streaming responses via Body::from_stream)

This avoids holding the full dataset in memory at any point.

Affected files

crates/domain/src/ports.rs — DiaryExporter port signature
crates/adapters/export/src/lib.rs — serialization implementation
crates/application/src/diary/export_diary.rs — use case
crates/presentation/src/handlers/diary.rs — HTTP handler
crates/adapters/sqlite/src/lib.rs — get_user_history query
crates/adapters/postgres/src/lib.rs — same

## Problem `diary/export_diary.rs` calls `get_user_history()` which loads the entire diary into a `Vec<DiaryEntry>` before serialization. For users with thousands of entries this holds everything in RAM at once. Current flow: 1. `DiaryRepository::get_user_history()` → `Vec<DiaryEntry>` (all in memory) 2. `DiaryExporter::serialize_entries(&[DiaryEntry], format)` → `Vec<u8>` (serialized bytes, also in memory) 3. Response written from the `Vec<u8>` ## Proposed fix Change the `DiaryExporter` port to accept a stream/writer instead of a slice, and have the repository yield rows lazily: - `DiaryRepository::stream_user_history()` → `BoxStream<DiaryEntry>` - `DiaryExporter::serialize_entries_stream(stream, format, writer)` — writes directly to an `AsyncWrite` or returns a `BoxStream<Bytes>` - The HTTP handler pipes the stream directly into the response body (Axum supports streaming responses via `Body::from_stream`) This avoids holding the full dataset in memory at any point. ## Affected files - `crates/domain/src/ports.rs` — `DiaryExporter` port signature - `crates/adapters/export/src/lib.rs` — serialization implementation - `crates/application/src/diary/export_diary.rs` — use case - `crates/presentation/src/handlers/diary.rs` — HTTP handler - `crates/adapters/sqlite/src/lib.rs` — `get_user_history` query - `crates/adapters/postgres/src/lib.rs` — same

GKaszewski commented

2026-06-10 09:22:26 +00:00

Constraint: the CSV column order (title,year,director,rating,comment,watched_at,external_metadata_id) must not change. Export and import are designed to be a round-trip — users can dump their diary and re-import it. Any streaming rewrite must preserve this exact format.

**Constraint**: the CSV column order (`title,year,director,rating,comment,watched_at,external_metadata_id`) must not change. Export and import are designed to be a round-trip — users can dump their diary and re-import it. Any streaming rewrite must preserve this exact format.

GKaszewski closed this issue

2026-06-11 23:20:57 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: GKaszewski/movies-diary#5