What are LLM evals?
LLM evals are repeatable tests that score the output of a large language model or AI agent against what a good answer or action looks like. Teams use them to measure quality, compare changes and catch regressions before users see them.
What is AI evaluation?
AI evaluation is the process of measuring whether an AI system does what it should, on the inputs it will actually see. LLM evals are the working tests inside that process: a set of inputs, a definition of a good result, and a way to score each output.
Evals matter because language models are non-deterministic and sensitive to small changes. A new prompt, a model upgrade or one added tool can fix ten cases and quietly break five others. Without evals, teams find out from users.
What types of LLM evals are there?
Most teams combine three kinds of scoring.
| Type | How it scores | Best for |
|---|---|---|
| Code-based checks | Exact match, rules, schema validation, "was this tool called with these arguments?" | Actions, formats, facts with one right answer |
| Model-graded (LLM as judge) | A second model scores the output against a rubric | Tone, helpfulness, faithfulness to a source |
| Human review | People rate or label outputs | Calibrating the other two, high-stakes cases |
Evals also differ by when they run. Offline evals run on a fixed test set before a change ships. Online evals score live traffic after release, usually on a sample, and feed AI observability dashboards.
For agents, score the actions as well as the words. An agent that writes a perfect reply but calls the wrong tool has failed.
How do you build an LLM eval set?
Start from real conversations, not invented prompts. Real requests carry the phrasing, typos and ambiguity your agent will face.
Say you run an agent that schedules meetings inside a CRM. A good starting set looks like this:
- Pull a few dozen real requests, including ones that went wrong: "book a call with Dana next week," "move my 3pm with Acme to Thursday," "set up a demo with the whole buying team."
- For each one, write down what correct looks like: which tool should be called, with which arguments, and what the agent should say or ask.
- Choose the scoring for each check. Tool and argument checks are code-based; the quality of the reply can be model-graded.
- Run the whole set on every change to the prompt, model or tools, and track the pass rate over time.
- Each time a new failure shows up with users, add it to the set so it never returns unnoticed.
If the agent isn't live yet, write the first set from support tickets, sales calls and the tasks the agent is meant to do, then replace it with real requests as they arrive.
Keep the set small enough to run often and representative enough to trust. A set that only tests easy requests will pass while users struggle.
How do evals fit into shipping an agent?
Evals turn "it seems fine" into a pass rate a team can track release to release. The strongest sets grow from live conversations, because those show what users really ask.
Foldspace's Trust Lab, in early access, is aimed at exactly this: teams will turn live conversations into scenarios the agent must handle correctly and run them on a schedule, so regressions show up before users see them.