Skip to content

Testing plan: the harness's GitHub verbs ​

Derived from requirements.md and design.md, before tasks.md. Authored at test-planning. The results section is filled at verification.

This file is executable content. The commands below are what the agent runs. Credentials appear by reference only. No test touches the network: the client is exercised over ghreplay's replaying connection, and core and the commands over FakeGitHubClient.

Test matrix ​

#TypeApplies?Scope / what it provesWhere it runs
T1Unit: the clientyestest_ghapi.py: each new method's exact verb, path, query and JSON body. The GraphQL reviewThreads document gets its variables and cursor walk tested. Hostile branch, SHA, method and coordinates are refused before any exchange. A 422 on create_pull and a 405 on merge_pull become GitHubApiErroruv run --project cli python -m pytest -q cli/tests/test_ghapi.py
T2Unit: coreyestest_github_ops.py: each operation over the fake client. Covered: data fields and exit codes; the comment published as comment.agent with record=True and a marked, enveloped body; the direct-post fallback when the bus raises; the checks roll-up table (success, failure, pending, none, statuses and runs mixed); resolved threads filtered; merge refused when mergeOnApproval: false; pr create defaulting base and repository, linking, and still exiting 0 when the link fails; --discover linking every PR found, and none found; the PR-argument grammaruv run --project cli python -m pytest -q cli/tests/test_github_ops.py
T3Unit: CLI commands and routingyestest_github_verbs_cli.py: argument groups (--body or --body-file, --pull-request or --discover), --body-file - from stdin, JSON output of show, status and threads, a detached HEAD refused, exit codes 0, 1 and 2. harness_routed: remote when healthy, local with the stderr note when not, never auto-starteduv run --project cli python -m pytest -q cli/tests/test_github_verbs_cli.py
T4Integration (scenario)yestest_github_verbs_integration.py, with Gherkin docstrings: a comment reaches the ledger and a subscribed fake channel, and the poller-side publisher then drops it as enveloped; a PR opened by pr create lands in the session registry through the same link_pull_request call sessions link-pr makes (its work-item-state.json write is issue-368's, already covered); the service's routes answer each operation through TestClient with the fake clientuv run --project cli python -m pytest -q cli/tests/test_github_verbs_integration.py
T5Contract / parityyestest_api_contract_parity.py: the authored OpenAPI file carries every new path, method and operationId for both roles; the MCP tool list is the same for both roles; test_standing_security_integration.py's tool list is updated; test_docs_parity.py: every new command has a pageuv run --project cli python -m pytest -q cli/tests/test_api_contract_parity.py cli/tests/test_standing_security_integration.py cli/tests/test_docs_parity.py
T6Unit: the hookyestest_harness_link_pr.py: git push, gh pr create, hub pull-request-shaped and MCP create_pull_request calls each run link-pr --discover with the resolved work item; the-loop pr create, an unrelated Bash call and an interrupted call run nothing; no response text reaches the argv; every path exits 0uv run --project cli python -m pytest -q cli/tests/test_harness_link_pr.py
T7Security / abuse caseyesone negative test per abuse case: the token is absent from a failing verb's output (A1); merge is refused under mergeOnApproval: false, and the route body cannot carry a policy (A2); hostile coordinates are refused before an exchange (A3); the comment is marked and enveloped (A4); the fallback note is printed (A5); the hook uses no shell and passes no response text (A6); ticket show fetches no attachment (A7)uv run --project cli python -m pytest -q cli/tests -k "abuse_447"
T8Performanceyes, boundedpr status makes exactly three exchanges on the replaying connectionpart of T1
T9End-to-endn/aa live GitHub exchange needs a token and a scratch repository this cloud session does not hold. Every process boundary (CLI to service to core to client) is covered by T3 and T4
T10Manual exploratoryn/asame reason as T9. The owner can run the-loop pr status github:MadaraUchiha-314/the-loop#<this PR> on a box with the daemon
T11Lint / format / typecheck / config validation / full suiteyesthe repository's own gates, as pre-commit and CI run themmake check
T12Security review (gate)yesthe-loop's checklist against A1–A7, recorded as evidence. Tier 3: the autonomous review sufficesevidence/security-review.md

Scenarios & requirement trace ​

RowRequirement(s)Scenario / case
T1R1.2–R1.8, NFReach request's exact exchange; refusals before a request
T2R1.1–R1.9, R4.1each operation's result, exit code and error path
T3R1.*, R2.1–R2.3, R4.1–R4.2the argument grammar; the routing choice and its note
T4R1.1, R1.4, R3.1Scenario: An agent's comment is recorded and mirrored, never re-published and its siblings
T5R3.1, R3.2, R5.2contract, tools and docs parity
T6R4.3, R4.4the trigger table
T7A1–A7one negative test each

Verification environment ​

The work item's cloud checkout: uv 0.12.x, Python 3.11, no gh on PATH, no token, and no network in tests.

Evidence to capture ​

  • evidence/verification.md: one section per row, with the command and the runner's summary line.
  • evidence/self-review.md, evidence/security-review.md, evidence/documentation.md, and evidence/reviewer-briefing.md (also posted on the PR).

Activities checklist ​

  • [x] T1 client
  • [x] T2 core
  • [x] T3 commands and routing
  • [x] T4 integration
  • [x] T5 parity
  • [x] T6 hook
  • [x] T7 abuse cases
  • [x] T11 make check
  • [x] T12 security review

Verification results ​

RowCommandOutcomeEvidence
T1pytest tests/test_ghapi.py73 passedverification.md § T1
T2pytest tests/test_github_ops.py83 passed§ T2
T3pytest tests/test_github_verbs_cli.py26 passed§ T3
T4pytest tests/test_github_verbs_integration.py tests/test_mcp_integration.py16 passed§ T4
T5pytest contract/docs parity15 passed§ T5
T6pytest tests/test_harness_link_pr.py34 passed§ T6
T7pytest -k abuse_44733 passedsecurity-review.md
T8part of T13 exchanges§ T8
T9, T10n/an/a (no token or scratch repository in the session)§ T9, T10
T11make check, step by stepgreen; 5211 passed, 1 skipped§ T11
T12the security checklistno unresolved findingsecurity-review.md

Released under the MIT License.