perf: import execute processes rows sequentially — slow for large imports with TMDB lookups #4

New Issue

GKaszewski · 2026-06-10T09:18:46Z

GKaszewski commented

2026-06-10 09:18:46 +00:00

Problem

crates/application/src/import/execute.rs processes each confirmed row in a sequential loop. For large imports where rows resolve to unknown movies (triggering a synchronous TMDB API call via MovieResolver), this can take minutes.

Example

500 rows, each needing a TMDB lookup (~200ms each) = ~100 seconds sequential vs ~10 seconds with concurrency 10.

Proposed fix

Process rows concurrently with a bounded semaphore (e.g. tokio::sync::Semaphore with limit 10), collecting per-row results into imported, skipped_duplicates, failed as today. Error collection needs care across tasks — use JoinSet or futures::stream::iter().buffer_unordered().

Affected files

crates/application/src/import/execute.rs

## Problem `crates/application/src/import/execute.rs` processes each confirmed row in a sequential loop. For large imports where rows resolve to unknown movies (triggering a synchronous TMDB API call via `MovieResolver`), this can take minutes. ## Example 500 rows, each needing a TMDB lookup (~200ms each) = ~100 seconds sequential vs ~10 seconds with concurrency 10. ## Proposed fix Process rows concurrently with a bounded semaphore (e.g. `tokio::sync::Semaphore` with limit 10), collecting per-row results into `imported`, `skipped_duplicates`, `failed` as today. Error collection needs care across tasks — use `JoinSet` or `futures::stream::iter().buffer_unordered()`. ## Affected files - `crates/application/src/import/execute.rs`

GKaszewski commented

2026-06-10 09:30:01 +00:00

Also affects: crates/application/src/integrations/confirm.rs — same sequential loop over confirmations, each calling review_logger.log_review() which can trigger a synchronous TMDB lookup for unknown movies. Same fix applies.

**Also affects**: `crates/application/src/integrations/confirm.rs` — same sequential loop over confirmations, each calling `review_logger.log_review()` which can trigger a synchronous TMDB lookup for unknown movies. Same fix applies.

GKaszewski closed this issue

2026-06-11 22:42:37 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: GKaszewski/movies-diary#4