Background Inference Roadmap

Overview

This document tracks the implementation plan for moving deep learning batch inference off the main thread so the UI stays responsive during long runs.

The plan has two stages: first, a simple fork-join approach where the worker accumulates all results and bulk-writes them after completion (Phases 1–2). Then, an upgrade to the write reservation + merge pattern where results appear progressively in the UI during inference (Phase 3). Both stages follow the Concurrency Architecture.

Design Rationale

This roadmap follows the thread confinement + merge model described in the Concurrency Architecture. DataManager stays single-threaded; the worker uses private data; results are merged on the main thread.

Why a separate MediaData instance?

VideoData wraps ffmpeg_wrapper::VideoDecoder, which holds stateful FFmpeg context (current decode position, codec context, frame buffer). If the worker thread seeks to frame N while the UI is displaying frame M, they corrupt each other’s state. Creating a second VideoData at the same file path gives the worker its own independent FFmpeg decoder — both can read frames concurrently from the same file without interference.

This aligns with ConcurrencyTraits<VideoData>::supports_cheap_clone = true — VideoData cannot be shared across threads but can be cheaply cloned. See Concurrency Architecture for the full access pattern decision matrix.

Why all-at-once results first?

For the initial implementation (Phases 1–2), the worker accumulates all results in a private buffer and bulk-writes them on the main thread after completion. This is the simplest correct approach: no synchronization, no merge timer, no partial-state concerns.

Phase 3 upgrades to the write reservation + periodic merge pattern from the Concurrency Architecture, giving progressive visibility — the user sees mask frames appearing as the model processes them.

Key Existing Patterns

MLCoreWidget PipelineWorker

src/WhiskerToolbox/MLCore_Widget/MLCoreWidget.cpp (L46–80):

• class PipelineWorker : public QThread with Q_OBJECT
• Emits progressUpdate(int, QString) signal from the worker thread
• Qt automatically delivers cross-thread signals via QueuedConnection
• Main thread connects QThread::finished to a lambda that harvests results and calls deleteLater()
• Panels disabled during execution via _setPipelineRunning(bool)

producer_consumer_pipeline

src/DataManager/transforms/Media/producer_consumer_pipeline.hpp:

• Uses std::mutex around MediaData access when multiple threads read frames
• Demonstrates that creating separate media access per thread is the safe approach

NotifyObservers::No

addAtTime(frame_idx, data, NotifyObservers::No) suppresses per-frame observer callbacks. Followed by a single notifyObservers() call after all data is written. Used to avoid N observer callback fires during bulk writes.

Implementation Plan

Phase 1: Worker Thread Infrastructure

• Define BatchInferenceResult struct — New code in SlotAssembler.hpp or a new header. Struct holding a vector of per-frame decoded results: FrameResult with frame index, decoded data variant, data_key, and decoder_id. Plus BatchInferenceResult with vector<FrameResult>, bool success, and string error_message. This is what the worker accumulates and the main thread consumes.
• Add runBatchRangeOffline() to SlotAssembler — Similar to the existing runBatchRange() but: (a) takes its own MediaData references instead of reading from DataManager, (b) returns BatchInferenceResult instead of writing to DataManager, (c) accepts std::atomic<bool> const & cancel_requested for cancellation. Uses a new decodeOutputsToBuffer() that returns decoded data instead of calling addAtTime(). Progress callback is still invoked per-frame.
• Create BatchInferenceWorker : QThread — Defined in an anonymous namespace in DeepLearningPropertiesWidget.cpp (following the PipelineWorker pattern). Constructor takes: SlotAssembler*, bindings, frame range, source image size, and a cloned MediaData. run() calls runBatchRangeOffline(). Emits progressChanged(int, int) signal. Stores BatchInferenceResult for harvest after completion. Holds std::atomic<bool> _cancel_requested checked in the loop.
• Clone MediaData for worker — In _onRunBatch(), create a new VideoData instance and call LoadMedia() with the same file path as the UI’s MediaData. Pass this private MediaData to the worker. The worker uses it for encoding; the UI continues using the original.

Phase 2: Wiring and UI

• Rewrite _onRunBatch() to async — Instead of calling runBatchRange() synchronously, create a BatchInferenceWorker and start it. Connect worker->progressChanged to the batchProgressChanged signal (no processEvents() needed — Qt queued connection handles it). Connect worker->finished to a lambda that: (1) harvests BatchInferenceResult,
  2. writes all results to DataManager on the main thread using addAtTime() with NotifyObservers::No, (3) calls notifyObservers() once on affected data objects, (4) calls worker->deleteLater(), (5) re-enables UI. Disable “Run Batch”, “Run Single”, and “Run Recurrent” buttons while running (following _setPipelineRunning() pattern).
• Add Cancel button — When batch starts, repurpose “Run Batch” to become “Cancel” (or show a separate button). On click, call worker->requestCancel() which sets _cancel_requested = true. The worker loop checks this before each frame and exits early. On cancellation, still write whatever results were accumulated up to that point.
• Remove processEvents() from batch progress — Since the event loop is now free (inference runs on a background thread), the processEvents(ExcludeUserInputEvents) call in the progress lambda is unnecessary. The progress bar updates naturally through queued signal connections.

Phase 3: Progressive Visibility via Write Reservations

Goal: Instead of waiting for the entire batch to finish, show results appearing frame-by-frame in the UI while the worker runs. This uses the write reservation + merge pattern from the Concurrency Architecture.

Prerequisite: createWriteReservation<T>() must be implemented on DataManager (see Concurrency Architecture Stage 2).

• Replace BatchInferenceResult buffer with write reservation — Instead of accumulating decoded results in a flat vector, the worker pushes results via a ResultCallback to a shared WriteReservation buffer. The WriteReservation class provides thread-safe push() and drain() methods protected by a mutex.
• Add merge timer to _onRunBatch() — Start a QTimer (200ms interval) when the worker launches. On each tick, call _mergeResults() which: (1) drains accumulated FrameResults from the WriteReservation, (2) writes them to data objects using addAtTime(..., NotifyObservers::No), (3) calls notifyObservers() once per affected key.
• Final merge on completion — In the finished handler, stop the merge timer, do one last _mergeResults() to pick up any frames computed since the last timer tick, then clean up the reservation.
• User editing during inference — With the reservation pattern, the user can edit masks at other frames in Media_Widget while inference runs. User edits go through getData() on the real object; worker writes go to the buffer. No conflict.

Phase 4: Testing and Documentation

• Unit test for cancellation — Test that runBatchRangeOffline() respects cancel_requested and returns partial results. Test that BatchInferenceResult correctly accumulates decoded data.
• Unit test for merge correctness — Verify that frames merged incrementally produce the same result as all-at-once delivery.
• Developer documentation — Create docs/developer/DeepLearning/background_inference.qmd documenting the threading pattern, rationale for separate MediaData, and how to generalize to other components.
• Update main roadmap — Update roadmap.qmd to reflect async inference progress.

Files to Modify

File	Changes
SlotAssembler.hpp	BatchInferenceResult struct, runBatchRangeOffline() declaration, ResultCallback type alias
SlotAssembler.cpp	runBatchRangeOffline() and decodeOutputsToBuffer() implementation, progressive callback support
DeepLearningPropertiesWidget.hpp	Worker pointer member, _setBatchRunning() method, QTimer* and WriteReservation members, _mergeResults()
DeepLearningPropertiesWidget.cpp	BatchInferenceWorker class, rewritten _onRunBatch(), cancel handler, merge timer, progressive merge

Files to Create

File	Purpose
WriteReservation.hpp	Thread-safe buffer for progressive result delivery

Verification

1. Build — cmake --build --preset linux-clang-release must succeed
2. Existing tests — All DeepLearning and SlotAssembler tests must pass
3. Manual — UI responsiveness — Start batch inference on >100 frames; verify Media_Widget slider and other widgets remain interactive
4. Manual — cancellation — Start batch, cancel mid-run; verify partial results are written and UI recovers cleanly
5. Manual — correctness — Compare batch output against single-frame inference for the same frames; results must match
6. Manual — video contention — During batch inference, scrub the media slider to confirm the UI’s video decoder works independently from the worker’s

Design Decisions

• All-at-once first, progressive merge second — Phases 1–2 use all-at-once delivery (simplest correct implementation). Phase 3 upgrades to write reservation + periodic merge for progressive visibility. This staged approach lets us ship a working background inference quickly and add polish later.
• Separate VideoData instance per worker — VideoData has supports_concurrent_read = false and supports_cheap_clone = true in ConcurrencyTraits. Cloning gives the worker its own FFmpeg decoder.
• QThread (not std::jthread) — Consistent with the existing MLCoreWidget pattern; integrates naturally with Qt signal/slot for cross-thread communication.
• Scope: deep learning only — DataTransform_Widget threading uses the same pattern but is out of scope for this plan.

Relationship to Concurrency Architecture

This roadmap implements the merge pattern from the Concurrency Architecture for the specific case of deep learning batch inference.

Architecture Concept	How It Applies Here
Thread confinement	DataManager writes only on main thread
ConcurrencyTraits	VideoData cloned (not shared); output data uses reservation buffer
Write reservation	Worker writes to private buffer; main thread merges periodically
Fork-join (Phases 1–2)	Worker returns BatchInferenceResult; main thread writes all at once
Merge (Phase 3)	Timer-driven merge for progressive visibility

The same pattern generalizes to DataTransform_Widget and any other long-running computation. See the architecture document for the full design.