MCP analytics: seeing what users do through an agent
When a user delegates to an agent, the session happens inside Claude or ChatGPT and your interface never sees it. What to capture.
A user says "file this expense" and it happens. No page loaded, no button clicked. Your product did the work and your analytics saw an API call with no session, no journey and no reason attached to it.
That gap is what MCP analytics fills.
What it is
MCP analytics captures and rebuilds the sessions where somebody works with your product through an AI client rather than your interface. Each session shows the user's intent, the agent's reasoning and every tool call, so you can see where tasks succeed and where they fail.
The unit is the session, not the request. A single user intent might be six tool calls, two of which failed and were retried differently. Six rows in a request log tell you nothing. One session tells you what happened.
Why the usual analytics cannot see it
Product analytics were built around a person and an interface: a page view, a click, a funnel. When a user delegates to an agent, none of those events exist.
| Human session | Agent session | |
|---|---|---|
| Where it happens | Your interface | Claude, ChatGPT, a coding agent |
| What you see | Page views, clicks, a funnel | API calls with no context |
| Who is acting | The user | An agent, on the user's behalf |
| Why | Inferable from the path | Only in the agent's own words |
| Failure looks like | An abandoned form | A successful call with a wrong outcome |
That last row is the important one.
The metric that misleads
Call-level success rate is the number everybody has and it is close to useless on its own.
An agent can make ten technically valid calls and still deliver the wrong outcome, because it misread a tool description, chose a tool that was not the right one, or interpreted an argument differently from how you meant it. Your success rate reads 100%. The user's task failed.
Measure task completion, not call success. A sequence of successful calls that left the user without what they asked for is a failure, and it is the most common kind.
What to actually measure
| Measure | What it tells you |
|---|---|
| Task completion per use case | Whether the product works through an agent at all |
| Tool call success per tool | A tool that fails often has a description problem, not a user problem |
| Tools never called | Either unnecessary, or described badly enough that no agent reaches for them |
| Argument errors per argument | Each one is a documentation defect with an address |
| Where sessions end | The last call before abandonment is your bug |
| Which client | Claude, ChatGPT, Claude Code, Codex, Cursor. They behave differently |
| Retries after a failure | An agent retrying the same call differently is telling you the error was unclear |
Group sessions by what the user was trying to do, then rank those groups by success rate. That ranking is your work queue, and it is usually short.
The thing this reveals that nothing else does
Tool descriptions that read well to a person often perform badly with an agent.
search_records: Searches records. is fine English and useless to a model deciding whether to call it. It says what the tool is and not when to use it. The version that works reads more like:
search_records: Find records by free text. Use this when the user describes
what they are looking for in their own words rather than giving an ID.
Returns at most 50 matches with their IDs; call get_record for the full
content of one.You cannot tell which of your descriptions is in which category by reading them. You can tell by watching the sessions.
Privacy, because it comes up immediately
An agent session contains what a user typed, which can be anything. Capture has to redact by default rather than on request: personal data and secrets stripped before anything reaches storage, not after.
That is a design constraint, not a setting. If redaction is optional, somebody will forget, and the thing they forget will be a customer's API key.
If you are comparing this with an LLM observability tool, Armature vs LangSmith, Langfuse and Braintrust sets out the difference: one watches the agent you build, the other watches the agent your user brought.
Where this sits next to discoverability
Two different problems that are easy to confuse.
Agent discoverability is whether an agent chooses your product at all. Measured in install share, before you have a user.
MCP analytics is what happens after. Whether the agent can use the product correctly, and whether the user's task succeeded.
Both matter and the second is where installs are lost after they are won. An MCP server that gets installed and then used badly gets removed, and the removal looks the same in your numbers as never being installed.
Getting started without buying anything
Log one thing per session: the tool calls in order, the arguments, the errors, and whether the user's stated intent was met. Even a rough version tells you which tool description to rewrite first.
Then read ten sessions by hand. In our experience the first ten are enough to find two defects, and both will be in a description rather than in the code.
Armature's self-serve product does this properly: the SDK wraps an MCP server, a Claude Connector or a ChatGPT App backend, rebuilds each session with intent and reasoning, groups them into use cases and issues, and redacts before storage. There are also eval suites that replay real workflows on every deploy so a regression is caught before a user finds it.
Common questions
What is MCP analytics?
Capturing and rebuilding the sessions where a user works with your product through an AI client rather than your interface. Each session shows the user's intent, the agent's reasoning and every tool call, so you can see where tasks succeed and where they fail.
Why do normal product analytics miss this?
Because the session happens inside Claude, ChatGPT or a coding agent, and your interface is never loaded. Your analytics see API calls with no session, no journey and no intent behind them.
What should you measure on an MCP server?
Task completion rather than call success, tool call success per tool, which tools are never called, argument errors per argument, and where sessions end. A sequence of successful calls that failed the user is still a failure.
Why does call-level success mislead?
An agent can make ten technically valid calls and still deliver the wrong outcome, because it misread a tool description or chose the wrong tool. Call success will read as 100% while the user's task failed.
Where this comes from
Armature ran 5,292 judged sessions with Claude Code, Codex and Cursor inside 51 realistic codebases, and published every run. The numbers on this page come from that work.