Testing plan: per-work-item model and effort choice, and spawning after the gate

Derived from requirements.md and design.md, before tasks.md. Authored at test-planning, completed at verification.

This file is executable content — it names commands an agent will run. No credential appears in it, by value or by reference: this work item reads and writes none.

What has to be proved

Three things carry the risk, and the matrix is shaped around them:

1. No comment text reaches an argv. Everything a reply can do is pick a key into the operator's declared list, and the argv is built from config or the adapter. This is the work item's one new trust boundary (T8).
2. Nothing runs on a model the harness refuses. The probe measures, the checklist withholds, resolution falls back, and the re-probe bounds the loop (T1, T2).
3. R8 cannot strand a work item. Deferral applies only at a human-gate start node; every other path spawns and respawns exactly as it does today (T2, T10).

Test matrix

#	Type	Applies?	Scope / what it proves	Where it runs
T1	Unit	yes	modelchoice resolution and merge order; modelprobe verdicts, digest invalidation and the withhold-not-introduce rule; adapter effort_args/with_args; selection.py rendering and per-section parsing; registry round-trip	make test (uv run --project cli python -m pytest -q cli)
T2	Integration (scenario)	yes	the four end-to-end behaviours below, Gherkin-documented, against a fake registry and a stub tmux	uv run --project cli python -m pytest -q cli/tests/test_*_integration.py
T3	Contract (OpenAPI)	yes	that the surface did not need a contract change: /api/v1/sessions types its response as untyped objects, so the three new record fields flow through and the contract-parity assertion still holds	make validate + test_api_contract_parity.py
T4	End-to-end	n/a — an end-to-end run needs a real harness, a real tmux server and a real GitHub repository. The verification environment section says why that is out of scope here and what stands in for it (T2 with stubs at the same seams).
T5	UI / visual	n/a — no product UI. The three human surfaces are text (checklist body, sessions list table, models check matrix) and are asserted as strings in T1/T2.
T6	Snapshot	n/a — the one rendered artifact that would justify a snapshot is the checklist body, and T1 asserts its rows individually, which fails more usefully than a whole-body diff.
T7	Performance / load	n/a — the feature adds no work to the delivery path: resolution reads a record the dispatcher already reads, and probing is off-path. T1 asserts the no-new-I/O property directly (no probe is invoked during a delivery).
T8	Security / abuse case	yes	one negative test per abuse case A1–A7 of requirements.md § Security considerations	uv run --project cli python -m pytest -q cli -k abuse
T9	Accessibility	n/a — no UI surface.
T10	Migration / upgrade	yes	a config with no new sections behaves exactly as today; a session record written before this change still parses; routing.harnessArgs still works behind the deprecation shim	make test
T11	Manual exploratory	n/a — every surface is deterministic and asserted; there is no interactive flow a human would find something in that T1/T2 would not.

Scenarios & requirement trace

Row	Requirement(s)	Scenario / case
T1	R2.5–R2.9, R3.1–R3.2	model name → [model_flag, name]; effort level → adapter.effort_args; merge order is base → model → effort; a harness with no model_flag offers no model section
T1	R2.2–R2.4	a bare name is a candidate everywhere; harnesses: narrows; an undeclared harness in harnesses: contributes nothing
T1	R7.2, R7.6	verdict caching, args_digest invalidation, unknown stays offerable
T1	R1.4–R1.5	per-section parse: one tick, no tick, two ticks, unknown token, ambiguous model beside a valid effort
T1	R5.1–R5.4	the three session fields round-trip; a record without them parses and renders -
T2	R1.3, R4.1–R4.2	Scenario: a work item that chose a model and an effort is respawned on both
T2	R7.3–R7.4	Scenario: a model the harness refuses is never spawned
T2	R8.1–R8.2	Scenario: an armed work item gets no session until its gate is answered
T2	R8.3	Scenario: a mid-graph work item still respawns
T8	A1	an unauthorized reply freezes nothing
T8	A2	a reply naming a flag, a path or a metacharacter resolves to no choice
T8	A3	the argv contains only what the config and the adapter produced
T8	A4	a hand-edited frozen record naming an undeclared choice is ignored
T8	A5	a label named after a model selects nothing
T8	A6	an unreadable checklist keeps the operator's arguments
T8	A7	a forged availability verdict can withhold a choice but never introduce one
T10	R6.1–R6.3, R2.12	an untouched config is byte-identical in behaviour; the routing.harnessArgs shim warns and works

Verification environment

• Repositories: this one only.
• Services / containers: none. Every external seam is stubbed — the harness binary (modelprobe takes an adapter, so the probe is a fake in tests), tmux (the existing runner stub), and the GitHub integration (the existing resolve("github", …) seam the selection-hook tests already patch).
• Fixtures & data: temporary directories for the registry, the portable state and the verdict cache; the shipped process graphs, unmodified.
• Credentials: none. This work item reads and writes no secret, token or environment variable, and the probe inherits the daemon's environment exactly as a critic run does.
• Bring-up: uv sync · Tear-down: none (pytest temporary directories).
• If bring-up fails: record it under Verification results, leave the dependent activities unticked, and escalate.

Why there is no T4. An end-to-end run would need a real harness CLI with real model access, a real tmux server and a real repository — none of which this repository's CI has, and a probe against a live vendor would make the suite depend on an account. T2 stands in for it at the same seams, which is where the logic being proved actually lives.

Evidence plan

Row	Evidence	Path under evidence/
T1	pytest summary (counts, duration)	unit.md
T2	scenario table (the-loop scenarios --format markdown) + run output	integration.md
T3	make validate output and the changed contract excerpt	contract.md
T8	the seven negative tests and their assertions	security-review.md
T10	the before/after sessions list table and the shim's warning	migration.md

Nothing captured here can contain a secret — no credential is read, and the only rendered output is a checklist body, a verdict matrix and a session table over fixture data. The redaction rule still applies to anything unexpected in captured output.

Verification activities

• [x] T1 — make test
• [x] T2 — uv run --project cli python -m pytest -q cli/tests/test_spawn_gate_integration.py cli/tests/test_dispatcher_choice.py
• [x] T3 — make validate (+ the two parity tests inside T1)
• [x] T8 — uv run --project cli python -m pytest -q cli -k abuse
• [x] T10 — uv run --project cli python -m pytest -q cli/tests/test_dispatcher_choice.py -k "before_this_feature or declared_nothing or deprecated"
• [x] lint/format/typecheck parity — ruff check, ruff format --check, pyright, markdownlint

Verification results

Executed on the work item's branch at 3c14f33. Full record: evidence/verification.md.

Activity	Command / procedure	Outcome	Evidence
T1 unit	make test	pass — 3603 passed, 1 skipped (3539 before this work item, so 64 new tests and none broken)	verification.md § T1
T2 integration	pytest cli/tests/test_spawn_gate_integration.py cli/tests/test_dispatcher_choice.py	pass — 19 passed; the four Gherkin scenarios named in the matrix	verification.md § T2
T3 contract	make validate + test_api_contract_parity.py, test_config_schema_parity.py	pass, and the row changed meaning: there was no OpenAPI edit to make	verification.md § T3
T8 security	pytest cli -k abuse	pass — 38 passed; the seven abuse cases, one new finding fixed	security-review.md
T10 migration	pytest cli/tests/test_dispatcher_choice.py -k "before_this_feature or declared_nothing or deprecated"	pass — 3 passed, plus the record-level half in test_routing.py	verification.md § T10
tooling parity	ruff check, ruff format --check, pyright, markdownlint	pass — the same commands CI runs	verification.md § Tooling parity

Not executed: T4 (end-to-end) stayed n/a as planned — no real harness, tmux server or repository in CI, with T2 standing in at the same seams. A live models check against a vendor was also deliberately not run: the probe spends real tokens and a suite depending on an account fails for reasons unrelated to the change, so test_modelprobe.py decides each outcome with a stub adapter instead.

Review comments

Appended by the-loop's record-feedback hook when a human gate approves with comments.