Armature vs LangSmith, Langfuse and Braintrust
LLM observability watches the agents you build. Armature watches the agents your users bring. Same word, opposite direction, and most teams need both.
These get compared because both contain the words "agent" and "observability". They watch systems pointing in opposite directions.
The short answer
LLM observability traces the agents your team builds, for the engineers who build them. Armature's MCP analytics captures the sessions where your users' agents use your product, for the product team. One watches your own agent. The other watches somebody else's agent using your service.
What LangSmith, Langfuse and Braintrust do well
They are good products solving a real problem, and it is not this one.
- Tracing your own chains and agents. Every step, prompt, token count and latency, in a system you control end to end.
- Prompt management and versioning. Which prompt version produced which behaviour.
- Evaluation of your own outputs. Datasets, scoring, regression checks on the agent you ship.
- Cost and latency. Where the tokens and the seconds go.
- Debugging. When your agent does something strange, the trace tells you why.
If you build an agent, you need one of these. That is not in question.
What they cannot see
The moment your product is used through somebody else's agent.
A user says "file this expense" in Claude. Claude reads your tool descriptions, decides which tool to call, formats the arguments, calls your MCP server, reads the response and decides what to do next. Then it reports back to the user.
You did not write that agent. You cannot instrument it. What reaches you is a series of requests.
| LLM observability | MCP analytics | |
|---|---|---|
| Whose agent | Yours | Your users' |
| Who runs the client | You | Anthropic, OpenAI, Cursor, whoever |
| Unit | A trace of your chain | A session of somebody else's agent using you |
| User intent | You logged it | Reconstructed from the session |
| Agent reasoning | Yours, fully visible | Reconstructed from what it did |
| Failure looks like | An error or a bad output | Ten successful calls and a wrong outcome |
| Audience | The engineers who build the agent | The product team who ship the service |
The failure mode that separates them
An agent can make ten technically valid calls to your server and deliver the wrong outcome, because it misread a tool description or picked a tool that was not the right one.
Your request log shows ten successes. Your uptime is fine. The user's task failed, they do not know why, and they blame your product.
Tracing your own agent cannot surface that, because the agent that failed is not yours. And a request log cannot surface it, because every request was valid.
What surfaces it is the session: what the user asked for, what the agent decided, what it called, and whether the thing the user wanted actually happened.
Where they overlap
Evaluation, partly. All of these tools can run a suite and score outputs.
The difference is what gets replayed. An LLM observability suite replays your prompts against your agent. An MCP eval replays a real user workflow through a real agent against your server, which is the thing that breaks when a model updates and your code did not change. See evals for an MCP server or a CLI.
Which you need
LangSmith, Langfuse or Braintrust if you build and ship an agent or an LLM feature. You need to see inside your own system.
MCP analytics if you expose an MCP server, a Claude Connector or a ChatGPT App, and users reach your product through a client you do not control.
Both if you do both, which a lot of companies now do. They observe different systems and neither substitutes for the other. Tracing your own agent tells you nothing about how a customer's Claude session used your tools, and vice versa.
And the third thing, which is neither
Before a user's agent can use your product badly, an agent has to choose your product at all.
That is agent discoverability, measured in install share, and it happens before any of the above has a session to look at. In our data, of the products that reached an agent's shortlist and were never adopted, four of the five recurring causes were integration defects rather than preferences.
Three questions, three measurements, in order: does an agent choose us, can it use us correctly, and does it still work after this change.
Common questions
What is the difference between LLM observability and MCP analytics?
LLM observability traces the agents your team builds, for the engineers who build them. MCP analytics captures the sessions where your users' agents use your product, for the product team. One watches your own agent; the other watches somebody else's using your service.
Do I need both LangSmith and MCP analytics?
If you build agents and also expose an MCP server, yes. They observe different systems. Tracing your own agent tells you nothing about how a customer's Claude session used your tools.
Can I use LangSmith to monitor my MCP server?
You can trace calls arriving at it, but you will see requests rather than sessions, and none of the agent's reasoning or the user's intent, because those happened in a client you do not run.
What does Armature do that an LLM observability tool does not?
It rebuilds the session from the other side: the user's intent, the agent's reasoning, every tool call, grouped into use cases and issues, from clients Armature does not control such as Claude, ChatGPT, Claude Code, Codex and Cursor.
Where this comes from
Armature ran 5,292 judged sessions with Claude Code, Codex and Cursor inside 51 realistic codebases, and published every run. The numbers on this page come from that work.