01What runs
A synthetic test is a user goal run by a real agent against your agent-facing interface. The target is an MCP server or CLI; it can be your existing interface or one Armature generated and hosts.
- Example workflow. Ask the agent to find the latest failed deployment, explain the likely cause, and cite the run it used.
- Example run. The agent searches deployments, opens the failed run, follows logs or status details, and returns the likely cause.
- Example evidence. Armature keeps the failed deployment ID, the tools the agent called, the log or status snippet it relied on, and the final explanation.
02How it runs
01
Connect the target
Register the MCP server or CLI that agents will use.
02
Run the workflow
Execute one workflow across the harness x model matrix.
03
Review the trace
Passing runs stay quiet; failures keep the evidence needed to repair the interface.
03Harness coverage
Verify the same workflow wherever a harness can actually run the model; unsupported pairs stay neutral, real failures stay visible.
Example workflow
Investigate a failed deployment
Model / harness
Claude Code
Codex
Cursor
OpenClaw
Gemini CLI
OpenCode
ChatGPT
Opus 4.8
✓
-
✓
✓
-
✓
-
Sonnet 4.6
✓
-
✗
✓
-
✓
-
Haiku 4.5
✓
-
✓
✓
-
✗
-
GPT-5.5
-
✓
✓
✗
-
✓
✓
Gemini 3.5 Flash
-
-
✗
✓
✓
✓
-
Kimi K2.5
-
-
✓
✓
-
✓
-
Qwen3.5 Coder
-
-
✓
✓
-
✗
-
Claude Code
Codex
Cursor
OpenClaw
Gemini CLI
OpenCode
ChatGPT
Opus 4.8
✓
-
✓
✓
-
✓
-
Sonnet 4.6
✓
-
✗
✓
-
✓
-
Haiku 4.5
✓
-
✓
✓
-
✗
-
GPT-5.5
-
✓
✓
✗
-
✓
✓
Gemini 3.5 Flash
-
-
✗
✓
✓
✓
-
Kimi K2.5
-
-
✓
✓
-
✓
-
Qwen3.5 Coder
-
-
✓
✓
-
✗
-