# AI observability

AI observability is the practice of recording and analyzing what an AI system does once it is live, including each step it took, its cost, latency, output quality and failures, so a team can explain its behavior, catch problems early and see where it falls short.

## What does AI observability track?

AI observability tracks six things for every request an AI system handles.

| Signal | What it captures | Question it answers |
|---|---|---|
| Traces | Each step: prompt, retrieved context, model calls, tool calls, final output | What did the system actually do? |
| Cost | Tokens and spend per request, user and feature | What is this costing us, and where? |
| Latency | Time per step and end to end | Where is it slow? |
| Quality | Eval scores, user ratings, human review | Were the answers and actions right? |
| Failures | Errors, timeouts, refusals, wrong tool calls | What broke? |
| Gaps | Requests the system couldn't handle | What do users need that it can't do yet? |

Traces are the foundation. A trace links every step of one request, so when an answer is wrong you can see whether the model misread the question, retrieved the wrong document or called the wrong tool.

## How is AI observability different from traditional observability?

Traditional monitoring asks whether the system is up and fast. With AI, a system can be up, fast and still wrong.

Take an AI agent in a billing app. A user asks, "switch our plan to annual billing starting next month." The agent calls the update-plan tool with the wrong start date, the API returns a success, and the agent confirms in a friendly sentence. Every infrastructure dashboard is green. Only a trace that shows the tool call and its arguments, checked against what the user asked, reveals the problem.

Three things make AI systems harder to observe:

- **Non-deterministic output.** The same input can produce different answers, so one passing test proves little.
- **Quality needs judgment.** "Correct" often depends on context, so teams combine automated evals, user feedback and human review.
- **Multi-step behavior.** Agents plan, call tools and retry, and an early mistake can surface many steps later.

## How do you get started with AI observability?

Instrument traces first, with cost and latency attached to each step. Then add a quality signal: a small set of evals on real requests, plus a thumbs-up or thumbs-down from users. Review failed and unresolved requests every week, and turn the recurring ones into fixes and new test cases.

For an in-product agent, the user's side matters as much as the trace: what people asked for, and whether they got it. Foldspace's Conversational Analytics, available now, covers that side: it [shows what each request to the Product Agent was trying to do](https://docs.foldspace.ai/whats-new/conversational-analytics/), how each conversation ended (resolved, unresolved or escalated), and the knowledge and action gaps worth fixing next.

## Learn more

- [Conversational Analytics](/platform/conversational-analytics)