AI agent observability: if you can't trace it, you can't trust it
AI agent observability: trace every run, approval and source, and track cost per step in USD so you always know what your agent did and exactly why it did it.
The problemYour agent made the wrong call and nobody in the company can explain why.
Friday, 5:15 p.m. A major client calls Javier, the head of sales, and he is upset: the quote he received that morning includes a 30% discount nobody authorized. The proposals agent sent it. Javier asks the obvious question: "Where did that discount come from?" The technical team digs for two hours, and the answer is uncomfortable: there is no way to know. The agent left no record of what it consulted or why it decided what it did. That is what happens without AI agent observability.
An agent you cannot explain is an agent you cannot defend in front of a client, an auditor or your own board. This article explains what it means to trace an agent, what data you need to record and how to turn that information into trust.
Why an agent with no audit trail is a business risk
Traditional software always does the same thing with the same input. If it fails, you review the code and find the bug. An AI agent does not work that way: it makes decisions on every run, consults different sources, calls tools and can reach different results from similar inputs.
That is not a flaw; it is exactly what makes it useful. But it has a consequence: if you do not record what happened on every run, there is no way to reconstruct it later. Looking at the final output is not enough. You need to see the path.
On top of that, failures do not always come from the agent's reasoning. Datadog's State of AI Engineering 2026 report found that 5% of model calls in production returned an error, and that capacity limits, rate limits and timeouts accounted for 60% of those errors. Without traces, an infrastructure error and a judgment error look the same from the outside: "the agent failed."
What AI agent observability really means
Observability means being able to answer four questions about any run, in minutes rather than hours:
- What did it do? The complete sequence of steps, in order, with start and end times.
- With what information? The sources it consulted: a document, a CRM record, a pricing rule, the client's memory.
- Who stepped in? What human approvals happened, who signed off and what changed before approval.
- What did it cost? The spend in USD for each step and for the full run.
If your system can answer those four questions, Javier's case gets solved in five minutes: you open the run and see that the agent read a pricing table from an expired promotion, and that the approval step was turned off for amounts below a certain threshold. Two concrete problems, two concrete fixes.
The four layers you need to trace
1. Every run
A run is one complete pass through a workflow: "generate a quote for client X." It needs a unique ID, a status (in progress, completed, failed, stopped by spending cap) and a record of all its steps. If your workflows coordinate several agents, the trace should show how the work was divided; we explain this in multi-agent orchestration.
2. Every approval gate
A gate is a point where the workflow pauses to wait for a person's sign-off. The trace must record who approved, when, which version they saw and whether they made changes. This protects the company and also the person who approved: it is clear what was in front of them when they decided.
3. Every event
Events are what happens inside a step: a model call, a tool query, a retry, an error, a rule that blocked an action. They are the fine-grained detail that lets you tell "the model didn't respond" apart from "the model responded with something wrong."
4. Every source
For every claim or data point the agent uses, you need to know where it came from. If the agent says "this client has preferred pricing," the trace has to show the exact record that says so. That is what turns an answer into something verifiable.
Cost per step in USD
This is the data point that changes the conversation with finance fastest. Instead of getting a monthly invoice from the model provider and trying to guess what drove it, every step records what it cost in dollars.
With that data you can answer questions that used to be impossible:
- Which workflow is the most expensive, and is it worth what it costs?
- Which client or type of request consumes the most?
- Which simple step is using a more powerful (and more expensive) model than it needs?
- What does a complete quote cost on average, compared with what it cost to do by hand?
That last question is the foundation of any serious return calculation. If you want to go deeper, we cover it in how to calculate the ROI of an agentic system.
Illustrative example: the quoting agent, before and after
Illustrative example. Before, Javier's team had a quoting agent that worked well most of the time. When it failed, someone opened the chat, read the response and tried to guess. Nobody knew what each quote cost. Client complaints arrived before any internal alert.
After instrumenting the workflow:
| Question | Before | After |
|---|---|---|
| Why was this discount sent? | Nobody knows | The trace shows the pricing table it consulted |
| Who approved the quote? | "I think nobody" | Record with name, time and version |
| Was it a model error or a system error? | Impossible to tell | Timeout event and model response, kept separate |
| What does a quote cost? | Unknown | Cost per step and total in USD |
The most valuable part was not resolving one incident. It was that, as traces accumulated, the team started spotting patterns: which types of requests triggered the most human corrections and which step concentrated the spend.
Common mistakes
- Logging only errors. Successful runs teach you something too: they define normal behavior so you can spot the abnormal.
- Storing everything with no structure. A huge text file is not observability. You need to filter by client, workflow, agent, date and cost.
- Forgetting the sources. Knowing what the agent said without knowing where it got it leaves the problem half solved.
- Letting only the technical team read the traces. The head of sales should be able to open a run and understand it without help.
- Tracing without protecting privacy. Traces contain sensitive data; they need the same permissions and controls as the rest of the system, as we explain in AI agent security.
From trace to improvement
Observability is not an end in itself. It is the raw material for improvement. Every traced run is a case you can review, turn into a test and use to measure whether the agent gets better or worse with each change. That cycle has a name, evals, and it is the natural next step after tracing.
What to do on Monday
Take the agent or automation that worries you most and ask your team the four questions about a run from last week: what it did, with what information, who stepped in and what it cost. If any answer takes more than a few minutes, you have found your first project.
At Addabra, every system we build traces every run, gate, event and source by default, with cost per step in USD, in a dashboard an executive can read, not just an engineer. It is not an add-on: it is part of the harness from day one.
Frequently asked questions
What is observability for AI agents?
It is the ability to reconstruct, step by step, what an agent did: what information it consulted, what it decided, who approved it and what it cost. Think of it as an airplane black box, except it is available at all times, not just after an accident.
Isn't saving the agent's conversations enough?
No. A conversation shows what the agent said, but not which sources it used, which tools it called, which rules it applied or what each step cost. Without that data you can read the error, but you cannot find its cause or keep it from happening again.
Why track cost per step instead of just the monthly total?
Because the monthly total tells you how much you spent, but not where. With cost per step in USD you know which agent, workflow or client consumes the most, and you can optimize right there, for example by using a cheaper model for a simple step.
