Capability: testing-and-contracts

How a work item is proved: a planned, gated testing plan and a verification node that executes it with committed evidence; integration-test coverage that is queryable (Gherkin scenario docstrings); and API contracts that are declarative (spec-first OpenAPI / GraphQL SDL).

What it is

The loop's answer to "is this actually done?" — planned before the code is written and gated after it is: testing-plan.md decides which kinds of testing apply to a work item, the verification node runs them and records the evidence, and two standing conventions make correctness self-describing (every integration test names the scenario it proves, every API is authored contract-first with docs generated from the contract).

Current behaviour

The testing plan and the verification node (issue-163)

• A work item SHALL carry a testing-plan.md in its spec folder, derived from design.md and locked at the test-planning phase — before tasks.md, because each task's _Test:_ names a row of the plan's matrix.
• The plan SHALL be reviewed at the same human gate as the design it derives from: test-planning sits between design and design-approval, so one approval covers both artifacts, feedback is recorded into each, and changes-requested returns to design, which re-derives the plan.
• The plan SHALL present a test matrix whose rows are candidate testing types (unit, integration, contract, end-to-end, UI/visual, snapshot, performance, security/abuse-case, accessibility, migration/upgrade, manual exploratory, plus anything the work item needs). No type is mandatory in itself — the matrix is work-item dependent — but every row SHALL carry a decision: a type that does not apply is marked n/a with a written reason. A trust boundary named in design.md §Security design SHALL have its negative test named here.
• The plan SHALL declare its verification environment — repositories, services, fixtures, bring-up/tear-down commands, and credentials by reference only (env var name / secret-store key, never a value). the-loop SHALL NOT introduce a runner, orchestrator or environment manager of its own: it facilitates verification by declaring what is needed and running the project's own commands, and links the operator's registered customInstructions docs rather than restating them.
• A testing plan names commands an agent will run, so it SHALL be reviewed as executable content, on the same footing as the operator's critics[] entries (decision-043).
• WHEN implementation completes THEN the verification node SHALL execute the plan and re-gate the same artifact: every activity ticked (checkmarks: complete) and a non-empty Verification results section recording, per activity, the exact command or procedure, the outcome and a link to the evidence.
• IF a planned activity cannot be executed THEN it SHALL NOT be ticked: the reason is recorded and the matrix replanned (with the reason) or the item escalated. IF the environment cannot be brought up THEN the loop SHALL escalate rather than pass the gate.
• Evidence SHALL be committed under docs/specs/<id>/evidence/ — a link to a CI run that expires or to a local path is not evidence. WHEN textual evidence is written THEN it SHALL be markdown (.md), never .txt — titled, one section per command, raw output in fenced blocks, and linted like every other markdown file; binary captures keep their own formats and are referenced from it. WHEN UI verification runs THEN it SHALL capture screenshots of each verified state and an animated capture (GIF or equivalent) when the behaviour under test is a flow rather than a state. Evidence SHALL be redacted (tokens, cookies, personal data, internal hostnames) before it is committed, because that directory is as public as the repository; a capture that cannot be redacted is not committed and the results row says so.
• The later evidence node SHALL summarise the verification results against the acceptance criteria rather than re-deriving them.

The process itself is integration-tested (issue-217)

• the-loop's own repository SHALL carry a scenario-driven end-to-end suite (cli/tests/test_pdlc_e2e_integration.py + cli/tests/test_pdlc_e2e/) that walks one work item per scenario through the shipped pdlc-work-item-loop via the real runtime and hooks, with the agent mocked as fixture playback and externals faked only at transport seams (the integrations.resolve labels/comments provider, the event-log path, the ask/reply comment poster and tmux runner). Authorization, loop-prevention filtering and every artifact gate run production code.
• Each scenario is a fixture set (a manifest of scripted steps — a closed vocabulary: comments, emitted artifacts, completion claims, ask/reply, inner-loop state, GitHub-outage knobs — plus expected trace); adding one requires a fixture directory and a one-line named test, kept in lockstep by a consistency test. A malformed set is refused naming the file and field, never silently skipped.
• The assertions are process conformance, not run-completion: the exact walked node sequence (skips marked distinctly, so a skip can never satisfy an expectation written for a pass), the exact loop:<phase> label trail, spec-chain artifacts locked before implementation is entered, expected events as an ordered subsequence of the log, and each gate's evidence/ record carrying the sections its node demands. Divergences report the first mismatch.
• The shipped scenarios pin: the full happy path (including await-inner-loops parking implementation until a simulated inner PR loop completes), the trivial-tier declared-skip short-circuit (with verification re-targeting evidence/verification.md per issue-179 and issue-365), ask/reply mid-flight (with the dead-pane reply refused fail-closed), gate rejection on an unlocked artifact, changes-requested looping back a phase, GitHub-outage degradation (graph.hook_degraded, verdicts unchanged), and loop prevention (marked and unauthorized comments never release a human gate).

Asynchronous tests wait on the state they depend on (issue-251)

• WHEN a test drives work onto a background thread THEN it SHALL wait on the state its next line depends on, and SHALL NOT wait on an earlier signal that merely tends to arrive first. A dispatch is an attempt (the spawn or delivery a test double records) and an outcome (the registry, dedup, event-log, announcement and graph writes made afterwards); the attempt is the visible one and the outcome is the one that matters.
• A fixed time.sleep before a positive assertion or a dependent action SHALL be read as a defect rather than a safety margin. A sleep guarding a negative assertion ("nothing should have happened") is a different construct and is permitted.
• WHERE the component under test offers a real barrier (draining and joining its workers) the test SHOULD take it and assert afterwards, rather than predicate-matching its way to the same point.
• the-loop's own suite SHALL carry pytest --dispatch-lag=<seconds>, which delays every dispatcher write that follows a spawn or a delivery so that a test waiting one step early fails on every run instead of about one in three. It SHALL be inert unless asked for, and its patches SHALL unwind per test.

A test reads no ambient machine state, and make runs what CI runs (issue-412)

• WHEN a test exercises code that resolves a path relative to the working directory or the home directory THEN it SHALL pin that resolution, and SHALL NOT let the answer depend on where pytest was started or on what the machine running it happens to have on disk. A test that passes from one directory and fails from another is not testing the behaviour it names.
• the-loop's CLI config is the standing instance of that shape: default_cli_config_path() falls through --config → $THE_LOOP_CLI_CONFIG → ./.the-loop/cli-config.yaml → ~/.the-loop/cli-config.yaml, and the last two branches read ambient state. A test touching it SHALL either pin the lookup (set $THE_LOOP_CLI_CONFIG to a path under its own tmp_path, leaving the real resolution to run and answer "no config" identically everywhere) or control it (chdir into a tmp_path and write the ./.the-loop/cli-config.yaml it means to read).
• A cwd-relative lookup is NOT ambient in itself, and SHALL NOT be pinned wholesale: the config path's parent anchors the state root, so a suite-wide pin moves that root out from under every test that legitimately controls the cwd branch. Fix the tests that neither pin nor control the lookup; leave the ones that do.
• RULE: a green CI is not evidence that the suite is hermetic — it is one row of the table. A suite is hermetic when it passes from any working directory, with and without the operator's own config present.
• WHEN the Makefile claims CI parity THEN each target SHALL run the command the corresponding pre-commit hook runs, working directory included, or its comment SHALL say where and why it differs. make test therefore runs cd cli && uv run python -m pytest -q, byte-for-byte the pytest hook's entry: running the same tests from the repo root is a different command wherever the-loop resolves anything cwd-relative, and a parity that is claimed but not kept sends the first contributor who hits it looking for a bug in their own diff.

A test controls the executables, paths and process metadata it asserts on (issue-454)

• WHEN a test asserts on what an executable lookup resolves (shutil.which, a harness binary's availability) THEN it SHALL fix that lookup itself, and SHOULD assert both answers. A developer with the Cursor CLI installed and a CI runner without it SHALL get the same verdict.
• WHEN a test asserts that a path stays inside a directory THEN it SHALL compare canonical paths on both sides. On macOS the temp dir /var/… is an alias of /private/var/…, so a resolved path never equals an unresolved one. The escape check stays: a hostile id SHALL still be shown to land directly inside the directory. A symlinked temp dir recreates the alias on any POSIX host.
• WHEN a test reads a process's session or process group THEN it SHALL use os.getsid / os.getpgid, never the sess column of ps. On BSD that column is a kernel session pointer, and macOS prints 0. Only columns every ps agrees on (ppid) are read from ps, and only where there is no /proc.
• WHEN a test needs a binary only to pass a preflight (the daemon's tmux check) THEN it SHALL get it from an explicit fixture rather than the host's PATH. preflight_tmux in cli/tests/conftest.py puts a stub tmux first on the daemon subprocess's PATH. The stub records any call and fails the test at teardown if one happened. A test that genuinely runs tmux SHALL declare a real dependency instead.
• make test SHALL pass on a supported Linux or macOS host whether or not tmux or any harness CLI is installed, and without a bespoke TMPDIR or PATH. A skip is allowed only where the platform cannot provide the capability under test.

The suite never writes into a checked-in .the-loop/ (issue-422)

• WHEN a test exercises code that writes state THEN it SHALL give that code a state root under tmp_path. A default relative to the working directory (RoutingConfig.portable_dir, StateLayout(), the event log's DEFAULT_PATH) or to the ambient config file (default_options() in either daemon) resolves to this repository's own .the-loop/ when pytest is started from the root. The write there is silent, and a staged-everything commit carries it in.
• The suite SHALL fail any test that writes into this repository's .the-loop/ or into the .the-loop/ of the directory pytest was started from. cli/tests/conftest.py enforces it with an audit hook that records each write-mode open, rename, mkdir, removal or link under either tree. An autouse fixture then fails the test and prints the path and the stack. Guarding the working directory's tree is what makes CI, which runs from cli/, enforce the rule too.
• The guard records and fails at teardown. It does not raise inside the hook, because fail-open code would swallow that and the test would pass. It sees the test process only, not its subprocesses.
• RULE: an assertion over a file SHALL establish that the file was written before it searches it. A "no secret in the log" check over a log that was never created passes without looking at anything.

Scenario docstrings and contract-first APIs

• Every integration test SHALL carry a Gherkin-syntax docstring (Feature: / Scenario: / Given-When-Then) naming the scenario under test, with a Requirement: link when tied to a spec's requirements.md (testing.gherkinDocstrings: required, testing.linkRequirements).
• Integration tests SHALL be discovered via testing.integrationTestGlobs.
• the-loop scenarios SHALL extract and tabulate all covered scenarios (--format table|markdown|json) so a harness or reviewer can query coverage.
• RESTful API specs SHALL be authored in specs/openapi/ (OpenAPI); GraphQL schemas SHALL be SDL-first under specs/graphql/; documentation SHALL be generated from those contracts, never hand-written (config.apiSpecs; not exercised in this repo — the-loop ships a CLI + docs, not an API).

Design

reference/testing.md · docs/specs/issue-163/design.md · docs/specs/issue-11/design.md · process-graph

History

Work item	What changed	Links
issue-454	The suite stopped reading the host it runs on (2026-10-02): on macOS four tests failed that Linux CI never saw — two critic-availability tests assumed cursor-agent was absent, the gate's containment test compared /var with /private/var, and the daemon detachment test read BSD ps's sess (a pointer, 0 on macOS) as the SID — and eight daemon tests failed wherever tmux was absent though none runs it. The tests now fix shutil.which, compare canonical paths (with a symlinked-temp-dir case that runs the alias on Linux), read os.getsid/os.getpgid, and take tmux from a preflight_tmux stub that fails any test which actually runs it	spec, issue
issue-422	The suite stopped writing into the repository it runs in (2026-09-22): four tests wrote into this checkout's .the-loop/ when run from the root, one of them into the tracked portable/, where an earlier run's control record had already been committed. Each now gets a tmp_path state root, the two debris records are deleted, and an audit-hook guard in conftest.py fails any test that writes into the repository's or the working directory's .the-loop/. Two of the four were weaker than they read: an abuse-case log check searched a file that was never written, and a lock assertion watched a pidfile the poller never held	spec, issue
issue-412	A test pins or controls what it resolves, and never reads the machine it runs on (2026-09-21): four test_instance.py argv assertions stopped passing or failing on the working directory and on whether the developer has ~/.the-loop/cli-config.yaml — pinned in the module that had the defect, not suite-wide, because the config path anchors the state root and a wholesale pin breaks the tests that legitimately control the cwd branch; and make test became byte-for-byte the pre-commit pytest hook, cd cli included, so the Makefile's CI-parity rule is true	spec, issue
issue-365	The e2e conformance keys followed the artifact (2026-09-14): executionLogSections and executionLogEntries became one evidenceSections map of file → sections, and the walk is asserted by the node trace alone rather than by checkpoints a hook wrote into a log	spec, decision-126, issue
issue-251	Waiting became a rule rather than a habit (2026-08-16): an asynchronous test waits on the state its next line depends on, not on the attempt that precedes it, and a fixed sleep before a positive assertion is a defect — with pytest --dispatch-lag=<seconds> shipped so the shape is found by running the suite rather than by reading it	spec, decision-091, issue
issue-217	The process itself became integration-tested (2026-08-12): a scenario-driven e2e suite drives one work item per scenario through the shipped outer loop against a fixture-playback agent, asserting process conformance (node trace with skips distinct from passes, label trail, locks before implementation, ordered event subsequence, execution-log sections) — seven scenarios covering the happy path with the inner-loop seam, trivial-tier declared skips, ask/reply with the fail-closed dead-pane refusal, gate rejection, review rejection looping back, GitHub-outage degradation, and loop prevention; new scenarios are fixture sets kept in lockstep with named tests by a consistency test	spec, issue
issue-163	Testing became part of the process rather than an assumption: the testing-plan.md artifact and the test-planning / verification nodes, the test-type matrix with n/a-with-a-reason, the declared-not-managed verification environment, and committed, redacted evidence (screenshots and GIFs for UI flows)	spec, decision-060, process-graph, issue
issue-165	Textual evidence is markdown (.md), never .txt — titled, one section per command, output in fenced blocks, linted like every other markdown file	spec, PR #168
issue-11	Introduced Gherkin scenario docstrings, the scenarios command and contract-first API conventions	spec, decision-014