AI agent monitoring measured against your own test set

LLMOps is the practice of running language model applications in production: tracing what they do, evaluating their outputs, versioning prompts, controlling cost and managing model changes. Our LLMOps glossary entry has the longer definition.

Normal application monitoring answers whether a service is up and fast. Datadog or Grafana will tell you the support assistant replied in 2 seconds with a 200 status code. Neither can tell you the reply quoted a refund policy you retired in March. An AI system can be fully available and still wrong, and nothing on an uptime dashboard turns red when that happens.

AI agent monitoring adds 4 things to the usual checks:

• Traces of every step: the prompt sent, the documents retrieved, the tools called and the final answer, so a bad answer can be followed back to its cause.
• Evaluations that score outputs against expected results, on a schedule and before every change.
• Cost attribution by feature, customer or agent, down to the token.
• A model lifecycle record: which model version runs where, when it retires and what replaces it.

We set these up on your accounts. Langfuse is open source and can run on your own infrastructure when conversation data cannot leave it. LangSmith suits teams already building with LangChain or LangGraph.

With an evaluation set: a fixed list of real cases from your business, each with the outcome a good answer must reach. We build the first one from 2 to 4 weeks of production traffic, chosen to cover common requests, awkward edge cases and past failures, then labeled by the people who know the right answer. Most sets start with 50 to 200 cases.

Each case is graded in the way that fits what correct means for that task:

The metrics depend on the system. For a RAG knowledge assistant we track whether the right document was retrieved and whether the answer stays inside it. For an agent that takes actions we track task completion, wrong tool calls and how often it hands off to a person. For a voice agent, completed bookings and transfers.

One accuracy percentage across everything hides the problems. We report by task, so you can see that billing questions score well while cancellation requests need work.

Mostly because the things around the model change. Your products, prices and policies move on, and the documents an assistant retrieves from fall out of date. Users start asking questions nobody planned for. Someone edits a prompt to fix one complaint and breaks 2 other cases. Tool APIs the agent calls change shape.

The models change too. Providers retire older versions on a published schedule. Anthropic commits to at least 60 days' notice before retiring a publicly released model, and on September 30, 2026 it announced that Claude Sonnet 4.5 retires on November 30, 2026 (Anthropic model deprecations (opens in a new tab)). OpenAI keeps a similar API deprecations page (opens in a new tab). API behavior moves as well: Anthropic's API version 2026-10-21 rejects requests that still set the temperature parameter, which many older integrations send by default.

Our rule for a new or replacement model is that nobody switches on launch day. We run the candidate against your evaluation set, compare accuracy, cost and latency with the current model, and read every case where their answers differ. If the candidate wins, we switch with a pinned model version and a tested rollback. If it loses on your cases, you stay put until the retirement date forces a decision, and by then you know exactly which prompts to rework.

Pinned versions matter. A moving model alias lets the provider change the model under you without any deploy on your side.

Plans cost $1,500 to $5,000 a month. One assistant with moderate traffic and a monthly evaluation run sits near $1,500. With 2 or 3 systems and weekly failure reviews, plans sit around $3,000. Several systems, high traffic, or regulated content that needs a person to review samples every week land near $5,000. Tracing platform fees and model usage are billed at cost with no markup, on your own accounts.

Cost work is part of every plan. These are the levers we check first:

OpenAI and Anthropic both price batch requests at about half their standard rate, and both discount cached input tokens. The first monthly report breaks cost down by feature before we change anything, so every saving is measured against a known starting point.

Your company is. In Moffatt v. Air Canada (2024), a British Columbia tribunal held the airline liable for wrong bereavement fare information its website chatbot gave a customer, and rejected the argument that the chatbot was responsible for its own actions (Moffatt v. Air Canada decision (opens in a new tab)). Treat anything your AI tells a customer as something your company said.

That shapes how we split the work. We define the checks, run them, flag failures and recommend fixes. You decide what error rate is acceptable for each task and which answers need a person to approve them first. For high-stakes outputs, such as refunds, medical or legal wording and anything that commits money, we design a human-in-the-loop step instead of trusting a score.

When AI ops monitoring is the wrong choice:

• Nothing is in production yet. Start with a build, such as custom AI agents, and tracing goes in from the first pilot.
• Traffic is tiny. Below about 300 conversations a month, a quarterly review costs less and catches the same problems.
• The trouble is plumbing. If triggers, connectors or credentials keep failing in n8n or Make, you need an automation care plan.
• You need a security assessment. We test for prompt injection inside the evaluation set, but we do not run penetration tests or cybersecurity audits.

Not included in any plan: overnight on-call cover, rebuilding the system and legal advice on AI regulation.