← All projects
2026Sole engineer — design, Rust implementation, tests
ADLC — adversarial verification judged by the filesystem, not the agent
The ADLC Execution Harness takes one seed MCP tool call and generates adversarial variants of it deterministically, offline, with no model in the loop. It captures a SHA-256 digest of every file under a root before and after the call, then judges the diff against a declared expectation — creates exactly these paths, no change, or no change outside an allowed set. It reports outcomes, never payloads.
RustAdversarial testingMCPState diffingContainment
8 strategiespath traversal, out-of-scope paths, oversized values, type confusion, null or empty, Unicode edge cases, numeric boundaries — and an unmutated control
Architecture
- 1Generate — mutate the seed call across eight strategies
- 2Prepare — snapshot the probe root: the workspace, or its parent for containment cases
- 3The host executes the call — the harness never does
- 4Judge — diff before and after against the declared expectation, exact-match
- 5Report — one JSON line per case: strategy, verdict, path counts; no arguments, no contents
The problem
Most approaches to agent safety infer what an agent did from what it said — traces, outputs, reasoning. A tool that overwrites a file with garbage and prints “done” looks identical to one that did the right thing. The only trustworthy witness is the state of the world before and after.
Approach & decisions
- Judge the world, not the agent. A verdict is “did exactly the right set of paths change”, computed from filesystem digests. The tool's output text is never consulted.
- Containment cases probe one directory above the workspace. A probe can only see what is under its own root, so a path-traversal test watching only the workspace could never see an escape — by construction, the escape lands somewhere the probe is not looking. The two containment strategies snapshot the parent instead, and an escape shows up as an unexplained create one level up.
- Comparisons are exact-match, not “at least contains”: a call that creates the expected file and two unexpected ones fails, because two unexpected creates is exactly what this exists to catch.
- Deterministic and offline. Eight rule-based strategies and no model key required — which is also why cargo test exercises the real generator rather than a fake. CaseGenerator is a trait so a model-backed generator can be a second implementation later.
- Fail-closed defaults: every non-containment mutation expects no change, because the harness cannot know whether a given tool should coerce a malformed argument and succeed. A tool that legitimately does gets a hand-written case with the expectation it actually deserves.
- Deliberately does not depend on Grit, even though both implement a small bounded-snapshot primitive — a documented trade-off that keeps each repository's CI independent.
Results
- Unit, integration and doc tests green in CI, with secret and dependency scanning on every push.
- The probe-above-the-workspace decision is a direct answer to the failure mode the UK AI Security Institute described in its August 2026 incident report: sandbox escapes during evaluation that were detected afterwards by general monitoring, with the margin between failure and success “resting on human vigilance rather than a technical barrier”.
- Honest scope: filesystem state only in this version. StateProbe is the trait a database or API probe would implement; neither ships yet, and neither is stubbed to look as if it does.
What this honestly is not
- Not a general correctness verifier. A verdict is about which paths changed, not whether their content is right. Bring your own content-level assertions.
- Not an executor. It brackets the call — prepare before, judge after — and the host runs it.
- Not an LLM. The strategies are a fixed, deterministic list.
Stack
Rustsha2walkdirserde_jsonclapGitHub Actions