The Compliant Neuron
Back to Lab
LIVE

Sandboxed AI Component Vetting

A static review, then each capability of a third-party AI component run alone in its own disposable sandbox, with fake credentials planted.

View on GitHub

MCP servers, plugins, agent skills, hooks and agent-driven apps all run with your permissions: they can read your files, use your network and start other programs. Reading their code tells you what they could do. This method adds evidence of what they actually do.

It extends our static-only AI Component Vetting skill with a runtime stage. Each capability — network, files, processes, credentials, starting at login, updating itself, and the text an agent will read — is run alone, in a disposable sandbox seeded with fake credentials (canaries). Anything that reaches beyond what that sandbox allowed is named, with the exact file, host or process.

It is model-neutral. Codex, Gemini CLI, Cursor, Claude Code, or a chat model handed the files: every agent starts at the same START-HERE file, checks what it can actually do, picks a mode to match, and works one stage at a time, stopping for a person's approval.

Four stages, four documents

  • Intake Is this exactly the thing that will be installed? Version, checksum, signature, source.
  • Static review What can it do, according to its code? Every capability, listed.
  • Probes What does each capability do when run alone in a sandbox? A person approves the probe plan before anything runs.
  • Report How risky is it, and what should be done? A scorecard, a risk tier and a concrete recommendation.

The scorecard

Five categories, each scored Pass, Partial or Fail, add up to a risk tier from Low to Critical:

  • Provenance Who made it, and can the copy be verified?
  • Permission scope Does it use only what its job needs?
  • Auditability Can its behaviour be inspected?
  • Blast radius What is the worst it could do if it is malicious or buggy?
  • Change control How does it change after you have trusted it once?

A component can only earn Low if every capability was actually probed. Every report has a Coverage Limitations section that says plainly what was not checked — and it is never empty.

Safety rules built in

  • A person approves first. Nothing third-party runs until a person adds an APPROVED line to the probe plan. The probe scripts refuse any probe that line does not cover, whichever AI is driving. It is a guardrail, not a lock: it proves the line exists, not who wrote it.
  • One capability per sandbox. Never shared, so one feature cannot hide behind another's permission.
  • No fallback to the real machine. Without a sandbox, a probe is reported as not run.
  • Content is data, not instructions. Everything inside the component — code, docs, tool descriptions — is treated as material under review, never as something to follow.

Worked example: VectorCraft 0.17.0

The repository includes a complete, real vetting of the VectorCraft vector editor and its built-in MCP server, from October 2026: 15 probes in Windows Sandbox, one fresh sandbox each, all within scope. With its root options set, the MCP server refused every out-of-root path tried. The result was Medium, install with mitigations — held there by the size of the code, which was spot-checked rather than read in full, and by its defaults.

Read the VectorCraft case study

Honest limits

  • Probes only see what happens during the probe. Logic triggered later, remotely, or by a feature nobody exercised can still hide.
  • Static review of a large codebase is a careful search, not a full read. Reports say which.
  • How well a run goes depends on the model driving it. The rules that matter most — approval, no run outside a sandbox — are enforced by the scripts, not left to the model.

This is due diligence, not a guarantee.

License & disclaimer — MIT

Open source, MIT-licensed, provided as-is. Due diligence, not a guarantee: probes only see what happens while they run. Review the method before relying on it in a regulated environment.