Static Input Capture & Preview (Phase 2)

Overview

Phase 2 of the input pipeline architecture adds Absolute capture mode for static (memory) model inputs. This allows users to encode a tensor from a specific frame and cache it for reuse across multiple inference runs, rather than re-encoding every time (Relative mode).

Architecture

CaptureMode Enum

Defined in DeepLearningBindingData.hpp:

  • Relative (default): Re-encodes data from DataManager at current_frame + time_offset every inference run. This is the original behaviour.
  • Absolute: Encodes data once at a user-chosen frame. The tensor is cached in SlotAssembler and reused on subsequent runs without re-encoding.

StaticInputData Extensions

Two new fields were added to the StaticInputData struct:

  • capture_mode_str: Serialization-friendly string ("Relative" or "Absolute")
  • captured_frame: The frame number that was captured in Absolute mode (-1 if not yet captured)

Helper methods captureMode() and setCaptureMode() convert between enum and string.

Tensor Cache in SlotAssembler

The SlotAssembler::Impl PIMPL struct now contains:

std::unordered_map<std::string, torch::Tensor> static_cache;

Keys use the format "slot_name:memory_index" (generated by staticCacheKey()).

Public API

Method Purpose
captureStaticInput() Encode data at a specific frame and store in cache
clearStaticCacheEntry() Remove a single cache entry
clearStaticCache() Clear all cached tensors
hasStaticCacheEntry() Check if a key exists in the cache
staticCacheKeys() List all cached keys
staticCacheTensorShape() Get tensor dimensions for preview
staticCacheTensorRange() Get min/max values for preview

Batch Dimension Expansion

When batch_size > 1 and the cached tensor has batch_dim=1, the tensor is expanded using torch::Tensor::expand() to match the requested batch size. This is a zero-copy operation.

Cache Invalidation

Cached tensors are invalidated in three scenarios:

  1. Model switch: resetModel() clears the entire cache
  2. Data source change: When the user changes the source combo for a static slot, the corresponding cache entry is cleared
  3. Re-capture: Calling captureStaticInput() overwrites any existing entry

UI Integration

Properties Widget

The static input group now includes:

  • Mode combo: Choose between Relative and Absolute capture modes
  • Time Offset spinbox: Enabled only in Relative mode
  • “Capture Current Frame” button: Enabled only in Absolute mode when the model is loaded. Encodes data at the current timeline position and caches it.
  • Capture status label: Shows whether a tensor is cached, which frame was captured, and the value range

View Widget

The View widget displays cached tensor information via refreshCachePreview():

  • Tensor shape
  • Value range (min, max)
  • Captured frame number
  • Cache status indicator

The preview is updated via the staticCacheChanged signal from the Properties widget.

Data Flow

1. User sets static slot to "Absolute" mode → offset spinbox disabled, capture button enabled
2. User navigates to desired frame in timeline
3. User clicks "Capture Current Frame"
4. _onCaptureStaticInput() calls SlotAssembler::captureStaticInput()
5. SlotAssembler encodes the frame data into a tensor and stores it in _static_cache
6. staticCacheChanged signal → ViewWidget::refreshCachePreview()
7. During inference, assembleInputs() finds cached tensor and uses it directly
8. For batch_size > 1, the cached tensor is expanded along the batch dimension

Serialization

The capture_mode_str and captured_frame fields are serialized via reflect-cpp as part of DeepLearningStateData. However, the actual cached tensors are not serialized — they must be re-captured after workspace restore. This is by design: tensor data can be large and the source data may have changed.