Command Architecture Roadmap

Progress Summary

Last updated: 2026-03-12

Phase	Status	Notes
Phase 1: Command Foundation	Complete	6 commands, factory, validation, variable substitution, recorder, mutation guard, golden traces
Phase 2: Triage Session	Nearly Complete	Core, widget, guided editor all done. Keyboard shortcuts (2.4) and tracked-regions overlay (2.5) remain.
Phase 3: Persistence & Save Registry	Complete	SaverRegistry, 8 savers (CSV ×6, CapnProto, OpenCV), SaveData + LoadData commands, round-trip fuzz tests
Phase 3.5: Command Recording & Test Generation	Not Started	Wire CommandRecorder into GUI, route widgets through executeSequence(), minimal Command Log UI
Phase 4: Action Journal Integration	Not Started	Generalizes CommandRecorder into persistent, timestamped journal
Phase 5: Undo/Redo	Not Started	CommandStack, memento-based EditorState undo, inverse commands
Phase 6: CLI Batch & Macro Recording	Not Started	Headless command execution; generalizes golden trace tests
Widget Command-ification	Ongoing	9 new commands identified via audit; none yet implemented. See GUI Verb Audit.

For the full architectural design, types, and code examples, see the Command Architecture Design Document.

What Exists Today

Registered Commands (6)

Command	Category	Undoable	Description
MoveByTimeRange	data_mutation	Yes	Move entities in a frame range between same-type data objects
CopyByTimeRange	data_mutation	No	Copy entities (source unmodified)
AddInterval	data_mutation	No	Append interval to DigitalIntervalSeries; creates series if missing
ForEachKey	meta	No	Iterate a list of values, execute sub-commands per item
SaveData	persistence	No	Save a data object via LoaderRegistry::trySave()
LoadData	persistence	Yes	Load a data object via LoaderRegistry::tryLoad()

Infrastructure

Component	Location	Purpose
CommandFactory	src/Commands/Core/	createCommand(), createCommandFromJson(), getAvailableCommands(), getCommandInfo()
CommandRecorder	src/Commands/Core/	Records executed commands for replay; hook in executeSequence()
MutationGuard	src/DataManager/Commands/	Debug-only: warns when DataManager is mutated outside a command
DataManagerSnapshot	src/DataManager/	Lightweight state snapshot (type, entity count, content hash) for golden trace tests
ParameterSchema	src/ParameterSchema/	Compile-time parameter schema extraction from reflect-cpp structs
AutoParamWidget	src/WhiskerToolbox/AutoParamWidget/	Dynamic Qt form builder from ParameterSchema
TriageSession	src/TriageSession/	Mark/Commit/Recall state machine using command sequences
SaverRegistry	src/DataManager/IO/core/	SaverInfo, AtomicWrite, getSupportedSaveFormats()
Validation	src/Commands/Core/	validateSequence() — structural, type, and variable checks
VariableSubstitution	src/Commands/Core/	Two-pass ${variable} substitution + normalizeJsonNumbers()

Test Coverage

• Command unit tests: 25+ test cases (factory, move, copy, interval, forEach, error handling)
• CommandRecorder tests: 19 test cases (record/clear, JSON serialization, executeSequence integration, MutationGuard)
• Validation tests: 10+ test cases (structural, deserialization, variable resolution, type compatibility)
• Variable substitution tests: 10+ test cases (static/runtime variables, integer normalization)
• SaveData/LoadData tests: 22 test cases (factory, execution, undo, serialization, round-trip)
• Saver fuzz tests: 9 fuzz test files (CSV round-trip for all 6 types, CapnProto lines, OpenCV masks)
• TriageSession tests: State machine transitions, runtime variable population, integration
• Golden trace tests: CopyByTimeRange, MoveByTimeRange, AddInterval, multi-command sequences, replay determinism

Dependency Graph (Remaining Work)

Completed Foundation (Phases 1, 3)
   │
   ├──► Phase 2 Remaining: Keyboard shortcuts (2.4), tracked-regions viz (2.5)
   │
   ├──► Phase 3.5: Command Recording & Test Generation
   │       Wires CommandRecorder into GUI, routes widgets through executeSequence()
   │       Enables: "do it in GUI → export JSON → write golden trace test"
   │
   ├──► Widget Command-ification (ongoing, on-demand)
   │       Uses: MutationGuard for gap detection, golden traces for verification
   │       Blocked commands: CreateData, DeleteData, AddEvent, RemoveEvent, etc.
   │
   ├──► Phase 4: Action Journal
   │       Depends on: Phase 3.5 (recorder wired into app)
   │       Generalizes recorder into persistent timestamped log
   │
   ├──► Phase 5: Undo/Redo
   │       Independent of Phases 2, 4, 6
   │       MoveByTimeRange already supports undo; needs CommandStack
   │
   └──► Phase 6: CLI Batch & Macro Recording
           Depends on: Phases 3 (done), 4
           Generalizes golden trace tests into CLI tool

Parallelizable: Phases 3.5, 5, and widget command-ification can proceed independently. Phase 4 depends on Phase 3.5. Phase 6 requires Phase 4.

---

Completed Phases (Summary)

Phase 1 — Command Foundation (Complete)

All 14 steps complete. Delivered:

• Core types (Steps 1.1–1.3): CommandResult, CommandContext, CommandDescriptor, CommandSequenceDescriptor, ICommand base class, CommandFactory with createCommand() / createCommandFromJson().
• Execution engine (Steps 1.4–1.5): Two-pass ${variable} substitution, executeSequence() with stop-on-error, SequenceResult for undo.
• Query helpers (Step 1.6): getEntityIdsInRange() for RaggedTimeSeries<T>, DigitalEventSeries, and DigitalIntervalSeries.
• First commands (Step 1.7): MoveByTimeRange, CopyByTimeRange, AddInterval, ForEachKey.
• Validation (Step 1.9): validateSequence() with structural, deserialization, variable, key existence, and type compatibility checks.
• GUI verb audit (Step 1.10): On-demand hybrid strategy. DataManager_Widget and DataInspector_Widget audited. See GUI Verb Audit.
• ParameterSchema (Step 1.11): Extracted to src/ParameterSchema/ as standalone library.
• CommandInfo (Step 1.12): getAvailableCommands(), getCommandInfo(), per-command ParameterUIHints.
• CommandRecorder + MutationGuard (Step 1.13): Trace recording with executeSequence() hook; debug-only strangulation pattern for runtime gap detection.
• DataManagerSnapshot + Golden Traces (Step 1.14): Lightweight snapshot comparison; golden trace test pattern for widget verification.

Known pitfall (resolved): rfl::Generic stores integers as double, causing round-trip failures. Fixed via normalizeJsonNumbers() in VariableSubstitution and createCommandFromJson() in CommandFactory.

See the design document for architecture details.

Phase 3 — Persistence Commands & Save Registry (Complete)

All steps complete. Delivered:

• SaverRegistry infrastructure (Step 3.1.1): SaverInfo metadata, AtomicWrite utility, getSupportedSaveFormats() on LoaderRegistry. See SaverInfo & AtomicWrite.

• 8 format savers (Steps 3.1.2–3.1.9):

  Format	Data Type	Fuzz Tested
  CSV	PointData	Yes (round-trip + delimiter)
  CSV	LineData (single-file)	Yes (round-trip + delimiter)
  CSV	LineData (multi-file)	Yes (round-trip)
  CSV	AnalogTimeSeries	Yes (round-trip + delimiter)
  CSV	DigitalEventSeries	Yes (round-trip + delimiter)
  CSV	DigitalIntervalSeries	Yes (round-trip + delimiter)
  CSV	MaskData (RLE)	Yes (round-trip + delimiter)
  CapnProto	LineData	Yes (exact binary round-trip)
  OpenCV (Image)	MaskData	Yes (robustness)

• SaveData command (Step 3.2): Dispatches via LoaderRegistry::trySave(). Not undoable. See SaveData Command.

• LoadData command (Step 3.3): Dispatches via LoaderRegistry::tryLoad(). Undoable (undo calls deleteData()). See LoadData Command.

---

Phase 2 — Triage Session (Nearly Complete)

Goal: Mark/Commit/Recall workflow using command sequences.

Completed steps:

• TriageSession core (Step 2.1): Idle ↔︎ Marking state machine in src/TriageSession/. commit() populates runtime variables (mark_frame, current_frame) and delegates to executeSequence(). Retains last commit commands for undo support.
• TriageSessionState (Step 2.2): EditorState subclass with pipeline/tracked-regions serialization.
• TriageSessionWidget (Step 2.3): Mark/Commit/Recall buttons, pipeline display, tracked-regions summary.
• Unit tests (Step 2.6): State machine transitions, runtime variable population, full triage round-trip integration test.
• AutoParamWidget extraction (Step 2.7): Extracted to src/WhiskerToolbox/AutoParamWidget/ with Qt6::Widgets + ParameterSchema dependencies only.
• Guided Pipeline Editor (Step 2.8): Command picker dialog, editable command list with AutoParamWidget rows, bidirectional sync with raw JSON editor.

Remaining Steps

Step 2.4 — Keyboard Shortcuts

• M — Mark (start a triage range)
• C — Commit (execute pipeline for [mark_frame, current_frame])
• R — Recall (abort, jump back to mark frame)

Step 2.5 — Tracked-Regions Visualization

• Render tracked-regions DigitalIntervalSeries as shaded overlay on the timeline/scrubber

---

Widget Command-ification (Ongoing)

Goal: Route all DataManager mutations through commands. Audited on-demand per widget.

Strategy: See GUI Verb Audit for the full audit process, widget tables, and command designs.

Definition of Done

A widget is fully command-ified when:

1. Zero direct DataManager mutations — all setData(), deleteData(), etc. go through createCommand() + execute().
2. Zero MutationGuard warnings — exercising the widget in a debug build produces no console warnings.
3. Golden trace test exists — at least one Catch2 test replays a recorded command sequence and asserts the final DataManagerSnapshot.

Audited Widgets

Widget	Verbs	Covered	New Cmds Needed	Golden Trace
DataManager_Widget	3	0	2 (CreateData, DeleteData)	Not yet
DataInspector_Widget	~30	~10	~14	Not yet

Commands Still Needed

9 commands identified by audit but not yet implemented. See New Commands Identified for full designs.

Wave 1 — Core CRUD: CreateData, DeleteData, AddEvent, RemoveEvent

Wave 2 — Entity Operations: DeleteByEntityIds, RemoveInterval, ModifyInterval

Wave 3 — Group Operations: AssignToGroup, RemoveFromGroup

Remaining widgets (DataImport, DataExport, Media, etc.) are audited on-demand as they are touched. MutationGuard provides automated runtime gap detection.

---

Phase 3.5 — Command Recording & Test Generation

Goal: Wire CommandRecorder into the running application so GUI actions produce exportable CommandSequenceDescriptor JSON, enabling semi-automated golden trace test generation.

Depends on: Phase 1 (done)

Motivation: All the infrastructure exists (CommandRecorder, executeSequence() hook, DataManagerSnapshot, golden trace tests) but nothing is connected in the running app. Widgets call cmd.execute(ctx) directly, bypassing executeSequence() and its recording hook. This phase wires everything together so a user can: do actions in GUI → see recorded commands → copy JSON → paste into a Catch2 golden trace test.

Current Gap

Component	Status
CommandRecorder class	✅ Implemented, tested (19 test cases)
executeSequence() recording hook	✅ Implemented (accepts CommandRecorder* param)
MutationGuard	✅ Active in debug builds
Shared recorder instance in app	✅ MainWindow owns CommandRecorder; commandRecorder() getter
ctx.recorder fallback	✅ executeSequence() falls back to ctx.recorder when no explicit recorder passed
DataSynthesizer “Generate”	✅ Routes through executeSequence()
SaveData call sites	✅ Routes through executeSequence()
TriageSession commit()	✅ Routes through executeSequence() with recorder
UI to view/export recorded commands	✅ CommandLog_Widget (Step 3.5.3)

Step 3.5.1 — Centralized Recorder (Complete)

• Add CommandRecorder* field to CommandContext (src/Commands/Core/CommandContext.hpp). Backwards-compatible default of nullptr.
• Update executeSequence() (src/Commands/Core/SequenceExecution.cpp) to use ctx.recorder as fallback when the explicit recorder parameter is nullptr.
• Add std::unique_ptr<commands::CommandRecorder> to MainWindow (src/WhiskerToolbox/Main_Window/mainwindow.hpp). Expose via commandRecorder() getter. Populate CommandContext::recorder for all widget command execution.

Step 3.5.2 — Route Widgets Through executeSequence()

• DataSynthesizer Generate (src/WhiskerToolbox/DataSynthesizer_Widget/UI/DataSynthesizerProperties_Widget.cpp): Build a CommandDescriptor{.command_name = "SynthesizeData", .parameters = ...} and call executeSequence() instead of cmd.execute(ctx).
• SaveData call sites (DataInspector savers, DataExport): Same pattern. See widget_save_refactoring_guide.qmd.
• TriageSession (src/TriageSession/TriageSession.cpp): Accept CommandRecorder* in commit(), forward to executeSequence().

Step 3.5.3 — Command Log UI (Complete)

• Create a minimal “Command Log” dock widget showing the list of recorded CommandDescriptor names and parameters from the shared recorder. See Command Log Widget.
• “Copy as JSON” button: calls recorder->toSequence("recorded_session"), serializes via rfl::json::write(), copies to clipboard.
• “Clear” button.
• Add “Modules → Command Log” menu item in MainWindow.

Note: This is intentionally minimal — NOT the full Phase 4 Action Journal. No persistence, timestamps, filtering, or provenance. Just enough to see what happened and export it.

Step 3.5.4 — End-to-End Verification

• Demo workflow: Open app → open Command Log → generate a SineWave via Data Synthesizer → save as CSV via DataInspector → Command Log shows 2 entries (SynthesizeData, SaveData) → “Copy as JSON” → paste into a Catch2 golden trace test with DataManagerSnapshot assertions → test passes.
• Debug build: exercising recorded workflows produces no MutationGuard warnings.
• Round-trip: executeSequence(recorded_json) then executeSequence(recorder.toSequence()) produce identical DataManagerSnapshots.

Exit Criteria

1. All GUI command execution paths go through executeSequence() with a recorder.
2. Command Log widget displays recorded commands in real-time.
3. “Copy as JSON” produces a valid CommandSequenceDescriptor that, when pasted into a Catch2 test and executed via executeSequence(), reproduces the same DataManagerSnapshot.

---

Phase 4 — Action Journal Integration

Goal: Commands auto-record to journal; journal can replay command entries.

Depends on: Phase 3.5 (recorder wired into app)

Generalizes the CommandRecorder into a persistent, timestamped, filterable log with workspace integration. The executeSequence() hook from Step 1.13 is reused — the change is swapping CommandRecorder* for ActionJournal* with persistence and metadata.

Step 4.1 — Journal Hook in executeSequence

• Generalize the CommandRecorder* parameter into ActionJournal*
• On successful command execution, record ActionEntry with timestamp, command name/description, serialized parameters, source widget

Step 4.2 — Journal Replay

• Deserialize ActionEntry::payload_json via createCommand() + execute() for replay
• Journal export: filter out undone entries, produce clean CommandSequenceDescriptor JSON

Step 4.3 — WorkspaceManager Integration

• Connect journal recording to existing WorkspaceManager provenance tracking

---

Phase 5 — Undo/Redo

Goal: CommandStack, EditorState memento undo, inverse-command undo for data mutations.

Depends on: Phase 1 (done)

Step 5.1 — CommandStack

• CommandStack class: push executed commands, undo/redo with CommandContext
• Clean state tracking: markClean() / isClean() for save-state awareness
• Redo stack cleared on new command push

See CommandStack in the design document.

Step 5.2 — WidgetStateChangeCommand

• Generic memento-based command for any EditorState
• Captures before/after JSON snapshots
• mergeWith() for coalescing rapid changes to the same widget

See WidgetStateChangeCommand in the design document.

Step 5.3 — CommandSequenceCommand

• Wraps vector<unique_ptr<ICommand>> for macro undo (e.g., triage commits)
• isUndoable() returns true if any child is undoable
• undo() reverses children in reverse order, skipping non-undoable ones

See CommandSequenceCommand in the design document.

Step 5.4 — MoveByTimeRange Undo Verification

• Integration test: move → undo → verify original state restored (MoveByTimeRange already reports isUndoable() = true)

Step 5.5 — UI Integration

• Edit → Undo / Redo menu items
• Ctrl+Z / Ctrl+Shift+Z keyboard shortcuts
• Integration with dirty tracking (EditorState::isDirty(), WorkspaceManager)

---

Phase 6 — CLI Batch Executor & Macro Recording

Goal: Execute command sequences headlessly; record GUI actions as commands.

Depends on: Phases 3 (done), 4

Generalizes the golden trace test pattern (load JSON → create DataManager → executeSequence() → check results) into a CLI tool. The macro recorder generalizes CommandRecorder from a test utility into a user-facing feature.

Step 6.1 — CLI Executor

• CLI tool or application mode: load a .json command sequence, create DataManager, execute, exit
• Variable substitution for batch parameterization (reuses existing pipeline variable system)
• Exit code reflects success/failure

Step 6.2 — Macro Recorder

• Intercept GUI actions (data loads, transforms, triage commits) and emit CommandDescriptors
• Record to an in-memory CommandSequenceDescriptor that can be saved to JSON

Step 6.3 — ConfigExportToolbar

• Reusable toolbar component: “Copy as JSON”, “Save Configuration…”
• Attach to widgets that produce command sequences (triage session, transform widget, etc.)

Step 6.4 — CommandSequenceWidget

• Standalone Qt widget (src/WhiskerToolbox/Commands/) for building, editing, and running command sequences outside the triage workflow
• JSON editor with validation feedback
• Command list view with run button

---

Relationship to State Management Roadmap

This roadmap replaces Phases 4–6 of the State Management Roadmap with a more comprehensive command-first architecture. The key differences:

State Mgmt Roadmap (old)	Command Architecture (this doc)
Phase 4: Action Journal records ad-hoc action types	Phase 4 here: records CommandDescriptors — standardized, replayable
Phase 5: Command interface with execute()/undo()	Phases 1+5 here: ICommand base with factory, CommandStack layered on top
Phase 6: SessionReplayer replays workspace provenance	Phase 6 here: CLI executor replays CommandSequenceDescriptor
No concept of triage or batch operations	Triage Session (Phase 2) demonstrates the architecture

Ideas carried forward from the State Management Roadmap:

• Action Journal (ActionEntry, append-only log, JSONL format, query/filter) — Phase 4
• CommandStack (undo/redo, clean state, merge) — Phase 5
• Memento-based EditorState undo (WidgetStateChangeCommand) — Phase 5
• Command merging for sequential similar operations — Phase 5 (ICommand::mergeWith())
• ConfigExportToolbar (“Copy as JSON”, “Save Configuration…”) — Phase 6
• Session replay / batch processing — Phase 6
• Incremental adoption (not every operation needs to be a command immediately) — design principle

See Also

• Command Architecture Design Document — full architecture, types, and code examples
• GUI Verb Audit — widget audit tables and new command designs
• Data Manager — core data management documentation