LLM evaluation (evals)

How it works

An eval set is a list of inputs paired with what a good output looks like. In a first version that is usually 50 to 300 real questions, documents or tickets, each with an expected answer, a required source or a pass rule. You run the system over the set and score every result, using the cheapest check you still trust:

• Exact checks: valid JSON, correct field values, the right tool called, a citation present.
• Model-graded checks, where a second model rates relevance or faithfulness to the source against a written rubric. This is often called LLM-as-judge.
• Human review of a sample, which keeps the model grader honest.
• Cost and latency per case, which matter as much as quality once volume grows.

Tools such as promptfoo, Langfuse, LangSmith and Braintrust run these suites and keep the history. A spreadsheet and a short script work for a first pass.

How businesses use it

Evals let you change a system without guessing. Before you switch models, rewrite a prompt or change chunk sizes in a RAG system, run the set and compare scores side by side. A cheaper model that scores the same on your cases is an easy decision. One that tops public benchmarks and scores lower on your invoices is a different conversation.

We build the first eval set during the Prove stage of every AI build, from the client's own historical tickets, documents or calls, and hand it over with the system.

Where it goes wrong

The common mistake is an eval set of easy, invented examples. It passes, and production still fails on the messy cases. Pull cases from real traffic, include the ones that went wrong, and add each new failure as a test.

Model graders have biases of their own, such as favoring longer answers or an answer in a particular position, as Zheng and colleagues (2023) (opens in a new tab) showed when they tested LLM-as-judge. Check a sample of their scores by hand every month. Keep the set stable, too: if it changes weekly, scores stop being comparable. Scoring live traffic on an ongoing basis belongs to LLMOps.

Do not buy tooling before you have a stable system to test. In the first week of a prototype, 20 cases in a spreadsheet are enough.