Skip to content

Verification: an operator can bring their own graphs and bind commands to them ​

The verification node's record, against testing-plan.md's matrix. Run in this container on the repository checkout (branch claude/github-issue-343-skvzmi, on top of 4cca7fa plus the review-round fixes committed with this record), with uv 0.12.18 installed over the image's own 0.8.17 (pyproject.toml requires >=0.12,<0.13).

Red first ​

The new tests were run against the tree before the implementation (the spec-only commit c100907, in a separate worktree, with the new test files copied in): tests/test_graph_catalog.py could not be collected (No module named 'the_loop.graph.catalog'), and of the other three files 21 failed, 42 passed — every new control, integration and scoped-attachment test failed; the passes were the pre-existing extension tests and the unauthorized-refusal scenario, which is a guard that must hold both before and after.

Results ​

#TypeCommandOutcome
T1Unitcd cli && uv run python -m pytest -q tests/test_graph_catalog.pypass — 83 tests in the file; T1's: every R1/R3 parse rule (grammar, reserved prefix, shipped names, duplicates, unknown keys, non-list and non-string values, command classes, double binding, reserved verbs), path resolution
T2Unitsame filepass — the compiler's errors on a custom graph, name mismatch, phase vocabulary (pinned equal to the shipped union), unreadable files, x- hook binding (declared, undeclared, no modules), the unknown-name error, slash_command, the repository-file warning
T3Unitsame filepass — resolve_outer_loop (7 cases), build_runtime choosing/refusing/guest, R4.4 (a broken declared graph raises), core.graphs.check on a custom item, _outer_loop_name state → record loop → binding → shipped, Runtime.start recording the custom name
T4Unitcd cli && uv run python -m pytest -q tests/test_control_custom_commands.pypass — 16: bindings, new-word parse, overrides, boundaries, ambiguity, case, keyword clash, the record's loop round trip and an older record
T5Integrationcd cli && uv run python -m pytest -q tests/test_custom_graph_integration.pypass — 3 Gherkin scenarios over a real dispatcher and a real git checkout: the-loop triage arms and the work item's state walks acme-triage-loop from triage; an overridden do walks acme-quick-loop; an unauthorized the-loop triage records, spawns and enters nothing
T6Unitcd cli && uv run python -m pytest -q tests/test_graph_extensions.pypass — 45 in the file; the new 4: scoped attachment skipped/applied, unknown and empty loops refused, a declared loop as a scope, digest changes
T7Unittests/test_graph_catalog.py -k loopspass — text and JSON rows, a broken graph exits 1, no module imported, an unreadable declaration and an unparseable config exit 1, attachment and x- checks, configured keywords, and a fresh-process run of the real CLI
T8Securitythe three files abovepass — each abuse case's named negative test (see security-review.md)
T9Contract / schemacd cli && uv run python -m pytest -q tests/test_config_schema_parity.py tests/test_configschema.py tests/test_docs_parity.pypass — 53: both schema copies identical, samples validate, every new schema leaf has a docs heading
T10Regressioncd cli && uv run python -m pytest -qpass — 4634 passed, 1 skipped, 180 s (baseline before the change: 4528 passed, 1 skipped)
T11Manual exploratory—not run — needs a GitHub repository and a running daemon; T5 drives the same dispatcher with the same comment against a real checkout
T12Documentation paritynpx markdownlint-cli2@0.18.1 over every changed pagepass — 0 errors over 17 files
T13Performance—n/a as planned — one cached compile per declared graph per process
T14UI—n/a as planned — no product UI

The rest of make check: uv run ruff check cli hooks — all checks passed; uv run ruff format --check cli hooks — 358 files already formatted; uv run pyright cli — 0 errors, 0 warnings.

What the run proves, in one line each ​

  • A graph the operator declares is compiled by the shipped rules and walked by the real runtime when an authorized user types the word bound to it (T2, T5).
  • An overridden shipped command walks the operator's graph, however the item was armed (T3, T5).
  • Nothing but the operator's current declaration can make a recorded name select a custom graph (T3, T8).
  • An operator who declares nothing sees no change (T10).

Re-run after the owner's reshape (PR #425 review) ​

The declaration moved to top-level graphs plus routing.control.commands. The same matrix was re-run on the reshaped tree:

#CommandOutcome
T1–T3, T7cd cli && uv run python -m pytest -q tests/test_graph_catalog.pypass: 105 tests. New: binding validation (undeclared graph, keyword on a built-in, malformed entries), the loader's routing._graphs fan-in, a binding to the default loop, a new word's own keyword in graph loops, and a binding to an undeclared graph reported by graph loops
T4cd cli && uv run python -m pytest -q tests/test_control_custom_commands.pypass: 23 tests, including a new word's keyword, a clash with a built-in keyword, re-pointing at a shipped loop, and a new word onto a shipped loop
T5cd cli && uv run python -m pytest -q tests/test_custom_graph_integration.pypass: the 3 scenarios. The dispatcher is now built from load_cli_config's output, so the fan-in is on the path
T9cd cli && uv run python -m pytest -q tests/test_config_schema_parity.py tests/test_configschema.py tests/test_docs_parity.py tests/test_onboarding_schema.pypass: the copies are identical, the new shape validates and routing.graph.graphs does not, every leaf has a heading, and graphs is in the execution onboarding group
T10cd cli && uv run python -m pytest -qpass: 4659 passed, 1 skipped, 185 s
T12npx markdownlint-cli2@0.18.1 over every changed pagepass: 0 errors

Also clean: uv run ruff check cli hooks, uv run ruff format --check cli hooks (358 files) and uv run pyright cli (0 errors).

Released under the MIT License.