Skip to content

Testing plan: multi-repo work items — the outer loop stays in the origin repo

Derived from requirements.md and design.md, reviewed with the design at one gate, and locked before tasks.md. Executed at the verification node, which fills in § Verification results below.

Test matrix

#TypeApplies?Scope / what it provesWhere it runs
T1Unityesrepo_state_key, inner_loop_state_dir, await_inner_loops (declared repos, both layouts), the phase-selection surface row (default, ticked, and never read as a phase), linked_work_items, render_assignment/render_graph_context surface + --pr-repo claimuv run --directory cli pytest -q
T2Integration (scenario)yesa cross-repo pull request routes to its work item, walks an inner loop whose state lands under the origin repo's spec directory, and holds the outer implementation gate until it completesuv run --directory cli pytest -q cli/tests/test_graph_multirepo_integration.py
T3Contract (OpenAPI)yesthe authored contract gains an optional prRepo on the five graph bodies and the graphShow query, and the served schema still matches it route-for-routeuv run --directory cli pytest -q tests/test_api_contract_parity.py
T4End-to-endn/a — an end-to-end run needs a live GitHub app and two real repositories; T2 exercises the same seams against the shipped router, runtime and hooks with no network.
T5UI / visualn/a — no user-facing surface (CLI + markdown).
T6Snapshotn/a — no rendered output is snapshotted in this repository.
T7Performance / loadn/a — the added work is one extra glob over a directory that holds one entry per pull request.
T8Security / abuse caseyesthe four abuse cases of design.md § Security design: path traversal from a payload, from --pr-repo, an unarmed work item reached by a cross-repo link, and a declared repo that never gets a pull requestuv run --directory cli pytest -q -k "traversal or pr_repo or unarmed or declared"
T9Accessibilityn/a — no UI.
T10Migration / upgradeyesan existing work item's pr-loops/pr-<n>/ keeps resolving unchanged, and a graph-state.json with no surface field reads as the default (the work item)uv run --directory cli pytest -q -k "back_compat or default"
T11Manual exploratoryn/a — every behaviour is reachable from a test; nothing here needs a human to look at it.
T12Parity (docs ↔ code ↔ schema)yesthe new config key is in the schema, in READS, and in docs/config/harness-config.md; every CLI flag is documenteduv run --directory cli pytest -q cli/tests/test_docs_parity.py cli/tests/test_harness_config.py cli/tests/test_graph_parity.py
T13Lint / type / formatyesruff, ruff format, pyright, markdownlint, and the harness-config schema validatormake lint typecheck (or the individual commands)

Scenarios & requirement trace

RowRequirement(s)Scenario / case
T1R1.3, R1.4, R1.6pr-loops/pr-7 for the origin repo, pr-loops/octo__infra/pr-7 for a contributing one; every malformed repo value raises
T1R2.1–R2.5, R2.7, R2.9the checklist offers the surface row; the surface resolves to work-item when untouched and pull-request when ticked; it is frozen into state and the frozen graph; an unticked row is neither a skip nor a refusal; the assignment and the prompt name it
T1R1.5a qualified closing keyword yields a ref in the other repository; an unqualified one stays local
T1R4.1–R4.4await-inner-loops: pass, wait-on-unfinished, wait-on-undeclared-loop, wait-on-declared-repo-with-no-loop, wait-on-unknown-origin
T2R1.1–R1.5, R4.1Scenario: a pull request in a contributing repository walks its own inner loop under the work item's spec directory
T2R4.2Scenario: the work item holds at implementation until every declared repository has finished
T8abuse cases 1–4negative tests named in design.md § Security design
T10R1.4, R2.3a pre-existing pr-loops/pr-<n>/ resolves unchanged; a state file with no surface reads as the default

Verification environment

  • Repositories: this repository only. The multi-repo behaviour is exercised with fixture directories and synthesised payloads — no second checkout, no network.
  • Services / containers: none.
  • Fixtures & data: tmp_path trees written by the tests; webhook payloads built in-test from the shapes already used in cli/tests/test_poller.py and test_graph_loops.py.
  • Credentials: none. No test in this work item authenticates to anything.
  • Bring-up: uv sync --directory cli · Tear-down: none (pytest tmp_path).
  • If bring-up fails: record it under Verification results, leave the dependent activities unticked, and escalate.

Evidence plan

RowEvidencePath under evidence/
T1, T2, T8, T10, T12full suite output (counts, duration) and the red→green transitionstests.md
T13ruff / ruff format / pyright / markdownlint / schema-validator outputlint.md
T2the scenario walk-through against the shipped graph and routermultirepo-scenario.md

Verification activities

  • [x] T1 — uv run --directory cli pytest -q (unit)
  • [x] T2 — uv run --directory cli pytest -q tests/test_graph_multirepo_integration.py
  • [x] T8 — the four abuse-case tests, red before the mechanism existed
  • [x] T10 — back-compat: existing pr-loops/pr-<n>/ and a config with no outerLoop
  • [x] T3 — uv run --directory cli pytest -q tests/test_api_contract_parity.py
  • [x] T12 — pytest -q tests/test_docs_parity.py tests/test_harness_config.py tests/test_graph_parity.py
  • [x] T13 — ruff check, ruff format --check, pyright, markdownlint, validate_config.py

Verification results

ActivityCommand / procedureOutcomeEvidence
T1uv run --directory cli pytest -qpass — 1524 passed, 1 skippedevidence/tests.md
T2uv run --directory cli pytest -q tests/test_graph_multirepo_integration.pypass — 6 passedevidence/tests.md
T8uv run --directory cli pytest -q -k "traversal or pr_repo or unarmed or declared"pass — 68 passed; red recorded before the mechanisms existedevidence/tests.md
T10uv run --directory cli pytest -q -k "back_compat or default"pass — 55 passedevidence/tests.md
T3uv run --directory cli pytest -q tests/test_api_contract_parity.pypass — 1 passed (folded into the T12 run after the review round)evidence/tests.md
T12uv run --directory cli pytest -q tests/test_docs_parity.py tests/test_harness_config.py tests/test_graph_parity.py tests/test_api_contract_parity.pypass — 32 passedevidence/tests.md
T13uv run ruff check cli hooks, ruff format --check cli hooks, pyright cli, markdownlint-cli2 "**/*.md", python scripts/validate_config.pypassevidence/lint.md
T2 (scenario)scripted walk of the ticket's own scenario against the shipped router, runtime and hookspassevidence/multirepo-scenario.md

Re-run after the PR #184 review round (the surface moved from the harness config to phase-selection): every row above is the re-run, plus pytest -q -k surface → 15 passed.

Not executed: none — every planned activity ran. One pre-existing tmux integration test (test_legacy_record_without_a_tmux_target_heals_via_respawn, untouched by this work item) failed once under the full suite and passed both in isolation and on a full re-run — a flake in this container, recorded here rather than smoothed over.

Review comments

Appended by the-loop's record-feedback hook when a human gate approves with comments.

Released under the MIT License.