# AI agent monitoring measured against your own test set

> AI agent monitoring is the work of tracing, scoring and tuning AI systems after they go live. We log every model call in Langfuse or LangSmith, build an evaluation set from your real cases, review failures weekly, test each prompt or model change before release and report accuracy and cost monthly. Setup takes 2 to 4 weeks. Plans cost $1,500 to $5,000 a month.

- Page: https://methodcrew.com/services/ai-ops-monitoring
- Last updated: 2026-10-28
- Publisher: [The Method Crew](https://methodcrew.com/)
- Service: AI ops monitoring and improvement
- Category: Care and retainers

We trace what your agents and assistants do, score their answers against cases from your own business and report accuracy and cost every month. When a provider retires a model, we test the replacement before you switch.

## Key facts

- Price: $1,500 to $5,000 per month. Set by the number of live AI systems, traffic and how much human review you need. Tracing tools and model usage are billed at cost. 3-month minimum.
- Timeline: 2 to 4 weeks to set up, then monthly
- Typical tools: Langfuse, LangSmith, Arize Phoenix, promptfoo, OpenTelemetry, OpenAI API, Anthropic API, Google Gemini API, Amazon Bedrock, Grafana, GitHub Actions
- Change requests: $175 an hour, quoted in writing first
- Model and tool usage: billed to your own accounts at cost, no markup
- Ownership: code, workflows, prompts, accounts and data belong to you

## Who it is for

- Companies with an AI agent, RAG assistant or voice agent in production, used by staff or customers
- Teams whose LLM API bill grew faster than usage and who cannot say why
- Product and operations owners facing a model retirement date from OpenAI or Anthropic
- Finance, healthcare and legal teams that need a written record of how AI answers are checked

## Signals you need it

- **Quality is judged by anecdote.** Someone forwards a bad answer in Slack, a prompt gets edited that afternoon, and nobody checks whether the edit broke something else.
- **Nobody knows the cost per conversation.** The monthly API invoice is one number. You cannot say which feature, customer or agent loop is driving it.
- **A retirement date is on the calendar.** Your provider has announced when the model you run will stop working, and the replacement has not been tested on your cases.
- **Prompts change straight in production.** System prompts live in a database field or a vendor dashboard, with no version history and no test before a change goes live.

## What we hand over

- **Tracing on your accounts.** Every model call, tool call and retrieval step logged with inputs, outputs, latency, tokens and cost, in Langfuse (self-hosted or cloud) or LangSmith.
- **Evaluation set.** 50 to 200 real cases with expected outcomes, graded by code checks, a calibrated model judge and a weekly human sample. It grows every month.
- **Quality and cost dashboard.** Accuracy by task, hand-off rate, cost per conversation and p95 latency, with alerts on cost spikes and error rates.
- **Release gate.** Every prompt, model or retrieval change runs the evaluation set in CI, using promptfoo or your tracing platform's evaluation runner, before it ships.
- **Model change log.** A dated record of each model version, prompt and setting in production, with the scores that justified every switch.
- **Monthly AI ops report.** Scores against the baseline, cost trend, incidents and their causes, changes shipped and any provider notices that affect you.

## How the engagement runs

1. **Map** (Week 1): We list every AI system in production: models, prompts, tools, data sources and the people who rely on the output. With the owners we write down what a correct answer looks like and what must never happen, such as quoting an unapproved price.
2. **Prove** (Weeks 1 to 2): Tracing goes live. We pull 2 to 4 weeks of real conversations, label a first evaluation set with your subject experts and score the current system. That baseline becomes the number every later change is measured against.
3. **Build** (Weeks 2 to 4): Dashboards, cost and error alerts, a release gate in CI and quick cost fixes such as prompt caching or routing simple steps to a smaller model. Guardrail checks for the failures you named during Map go in here.
4. **Run** (Monthly): Weekly failure review, monthly evaluation runs and report, prompt tuning, and testing of new or replacement models before any switch. Every bad answer your staff flag becomes a permanent test case.

Every engagement follows [The Method](https://methodcrew.com/approach): Map, Prove, Build, Run.

## What is LLMOps, and how is it different from normal monitoring?

LLMOps is the practice of running language model applications in production: tracing what they do, evaluating their outputs, versioning prompts, controlling cost and managing model changes. Our [LLMOps glossary entry](https://methodcrew.com/glossary/llmops) has the longer definition.

Normal application monitoring answers whether a service is up and fast. Datadog or Grafana will tell you the support assistant replied in 2 seconds with a 200 status code. Neither can tell you the reply quoted a refund policy you retired in March. An AI system can be fully available and still wrong, and nothing on an uptime dashboard turns red when that happens.

AI agent monitoring adds 4 things to the usual checks:

- Traces of every step: the prompt sent, the documents retrieved, the tools called and the final answer, so a bad answer can be followed back to its cause.
- Evaluations that score outputs against expected results, on a schedule and before every change.
- Cost attribution by feature, customer or agent, down to the token.
- A model lifecycle record: which model version runs where, when it retires and what replaces it.

We set these up on your accounts. Langfuse is open source and can run on your own infrastructure when conversation data cannot leave it. LangSmith suits teams already building with LangChain or LangGraph.

## How do you measure whether an AI agent is accurate?

With an evaluation set: a fixed list of real cases from your business, each with the outcome a good answer must reach. We build the first one from 2 to 4 weeks of production traffic, chosen to cover common requests, awkward edge cases and past failures, then labeled by the people who know the right answer. Most sets start with 50 to 200 cases.

Each case is graded in the way that fits what correct means for that task:

| Grader | Good for | Limits |
|---|---|---|
| Code checks | Valid JSON, the right tool called, required fields present, banned phrases absent | Cannot judge tone or reasoning |
| Model judge | Whether an answer is supported by the retrieved documents, follows policy and answers the question asked | Must be checked against human labels; we report how often they agree |
| Human review | High-stakes answers, new failure types and a weekly random sample | Slow and costly, so used where it counts |

The metrics depend on the system. For a [RAG knowledge assistant](https://methodcrew.com/services/rag-knowledge-systems) we track whether the right document was retrieved and whether the answer stays inside it. For an agent that takes actions we track task completion, wrong tool calls and how often it hands off to a person. For a voice agent, completed bookings and transfers.

One accuracy percentage across everything hides the problems. We report by task, so you can see that billing questions score well while cancellation requests need work.

## Why do AI systems get worse over time, and what happens when a model is retired?

Mostly because the things around the model change. Your products, prices and policies move on, and the documents an assistant retrieves from fall out of date. Users start asking questions nobody planned for. Someone edits a prompt to fix one complaint and breaks 2 other cases. Tool APIs the agent calls change shape.

The models change too. Providers retire older versions on a published schedule. Anthropic commits to at least 60 days' notice before retiring a publicly released model, and on September 30, 2026 it announced that Claude Sonnet 4.5 retires on November 30, 2026 ([Anthropic model deprecations](https://platform.claude.com/docs/en/about-claude/model-deprecations)). OpenAI keeps a similar [API deprecations page](https://developers.openai.com/api/docs/deprecations). API behavior moves as well: Anthropic's API version 2026-10-21 rejects requests that still set the temperature parameter, which many older integrations send by default.

Our rule for a new or replacement model is that nobody switches on launch day. We run the candidate against your evaluation set, compare accuracy, cost and latency with the current model, and read every case where their answers differ. If the candidate wins, we switch with a pinned model version and a tested rollback. If it loses on your cases, you stay put until the retirement date forces a decision, and by then you know exactly which prompts to rework.

Pinned versions matter. A moving model alias lets the provider change the model under you without any deploy on your side.

## What does AI ops monitoring cost, and how do you cut LLM API spend?

Plans cost $1,500 to $5,000 a month. One assistant with moderate traffic and a monthly evaluation run sits near $1,500. With 2 or 3 systems and weekly failure reviews, plans sit around $3,000. Several systems, high traffic, or regulated content that needs a person to review samples every week land near $5,000. Tracing platform fees and model usage are billed at cost with no markup, on your own accounts.

Cost work is part of every plan. These are the levers we check first:

| Lever | Where it helps | What to watch |
|---|---|---|
| Prompt caching | Long system prompts or documents reused across many calls | Saves money only while the cached prefix stays identical |
| Batch API | Work that can wait: document backlogs, report drafts, evaluation runs | Results can take up to 24 hours, so never for live chat |
| Smaller model for simple steps | Classification, routing and field extraction | Needs its own evaluation run before the switch |
| Less retrieved context | RAG systems that send 10 or more chunks per question | Too little context lowers answer quality, so measure both |
| A cap on agent steps | Agents that loop on tool calls | A cap set too low ends tasks early |

OpenAI and Anthropic both price batch requests at about half their standard rate, and both discount cached input tokens. The first monthly report breaks cost down by feature before we change anything, so every saving is measured against a known starting point.

## Who is responsible when an AI system gives a wrong answer?

Your company is. In Moffatt v. Air Canada (2024), a British Columbia tribunal held the airline liable for wrong bereavement fare information its website chatbot gave a customer, and rejected the argument that the chatbot was responsible for its own actions ([Moffatt v. Air Canada decision](https://decisions.civilresolutionbc.ca/crt/crtd/en/525448/1/document.do)). Treat anything your AI tells a customer as something your company said.

That shapes how we split the work. We define the checks, run them, flag failures and recommend fixes. You decide what error rate is acceptable for each task and which answers need a person to approve them first. For high-stakes outputs, such as refunds, medical or legal wording and anything that commits money, we design a [human-in-the-loop](https://methodcrew.com/glossary/human-in-the-loop) step instead of trusting a score.

When AI ops monitoring is the wrong choice:

- Nothing is in production yet. Start with a build, such as [custom AI agents](https://methodcrew.com/services/custom-ai-agents), and tracing goes in from the first pilot.
- Traffic is tiny. Below about 300 conversations a month, a quarterly review costs less and catches the same problems.
- The trouble is plumbing. If triggers, connectors or credentials keep failing in n8n or Make, you need an [automation care plan](https://methodcrew.com/services/automation-care-plans).
- You need a security assessment. We test for prompt injection inside the evaluation set, but we do not run penetration tests or cybersecurity audits.

Not included in any plan: overnight on-call cover, rebuilding the system and legal advice on AI regulation.

## What changes for your team

- An accuracy score per task, measured on your own cases, that you can compare month to month
- A known cost per conversation or per document, with the biggest cost drivers named
- Replacement models tested on your evaluation set well before the old model's retirement date
- Every prompt and model change passes a release gate and leaves a written record
- Bad answers flagged by staff land in a review queue and become permanent test cases

## Tools we work with

Langfuse, LangSmith, Arize Phoenix, promptfoo, OpenTelemetry, OpenAI API, Anthropic API, Google Gemini API, Amazon Bedrock, Grafana, GitHub Actions

## What it costs

$1,500 to $5,000 per month. Set by the number of live AI systems, traffic and how much human review you need. Tracing tools and model usage are billed at cost. 3-month minimum.

- Change requests: $175 an hour, quoted before work starts
- Model and tool usage: billed at cost, no markup

All 20 price bands: https://methodcrew.com/pricing

## Blueprints that use this service

- [Support knowledge assistant](https://methodcrew.com/blueprints/support-knowledge-assistant): built in 6 to 12 weeks, $20,000 to $60,000 per project.
- [AI receptionist voice agent](https://methodcrew.com/blueprints/ai-receptionist-voice-agent): built in 5 to 7 weeks, $12,000 to $30,000 per project.

## Common questions

### Can you monitor an AI system someone else built?

Yes. We add tracing through the provider's SDK or an OpenTelemetry exporter, which usually means a small code change your developers can review and deploy. If the system is a closed vendor product with no access to prompts or logs, we can still run an evaluation set against it from the outside, but step-level tracing and cost attribution will not be possible.

### Do you need access to our customer conversations?

We need enough real traffic to build the evaluation set. Traces stay on your accounts, and Langfuse can run on your own infrastructure if data cannot leave it. Where your policy requires it, we redact names, emails, phone numbers and account numbers before traces are stored, and we sign a data processing agreement before setup starts.

### Which models and providers do you support?

OpenAI, Anthropic and Google Gemini models through their own APIs, models hosted on Amazon Bedrock, and open-weight models you run yourself, as long as we can capture inputs and outputs. The evaluation approach is the same for all of them. When a retirement date or a price change forces a decision, comparing providers on your own cases is part of the plan.

### Can a model judge be trusted to grade answers?

Only after it has been checked. We label a sample of answers by hand, run the judge on the same sample and measure how often the judge and the human labels agree. If agreement is low for a task, we rewrite the judge's instructions or keep that task on human review. The monthly report states the agreement rate for each task, so you know how much weight each score can carry.

### How quickly do you respond when an AI system misbehaves?

Within 4 business hours for a serious incident, such as an agent quoting wrong prices or showing data it should not. Every system we monitor has a documented fallback in its runbook: switch the feature off, route to a person or roll back to the previous prompt. Your team can trigger it without us. We work business hours and do not offer overnight cover.

### Will tracing slow our system down?

Not in a way users notice. The Langfuse and LangSmith SDKs send traces in the background, so the user does not wait for logging. Model-judge evaluations run on a schedule or in CI, never inside a live request. The one check that can add latency is a guardrail placed before an answer is shown, and we only add those where the risk justifies the wait.

## Related services

- [Custom AI agents and copilots](https://methodcrew.com/services/custom-ai-agents): Agents that look things up in your CRM or helpdesk and take the next step, with approvals. $15,000 to $90,000 per project.
- [RAG and knowledge systems](https://methodcrew.com/services/rag-knowledge-systems): Your documents turned into an assistant that cites its sources. $15,000 to $75,000 per project.
- [Automation care plans](https://methodcrew.com/services/automation-care-plans): Monitoring, fixes and small changes for the automations you rely on. $1,500 to $6,000 per month.

## Further reading

### Guides

- [How to build a company knowledge assistant that cites its sources](https://methodcrew.com/insights/build-rag-knowledge-assistant)
- [How to measure the ROI of AI automation, running costs included](https://methodcrew.com/insights/measure-ai-automation-roi)

### Terms explained

- [AI guardrails](https://methodcrew.com/glossary/ai-guardrails): AI guardrails are checks placed around a language model that block, rewrite or escalate inputs and outputs that break your rules on safety, accuracy, privacy or scope.
- [AI hallucination](https://methodcrew.com/glossary/ai-hallucination): An AI hallucination is output from a language model that is fluent and confident but false, such as an invented citation, policy, figure or product detail.
- [LLM evaluation (evals)](https://methodcrew.com/glossary/llm-evaluation): LLM evaluation scores a language model system's outputs against a fixed set of real test cases, so you can tell whether a change made it better or worse.
- [LLMOps (AI observability)](https://methodcrew.com/glossary/llmops): LLMOps is the operating discipline for language model systems in production: tracing every call, tracking cost and latency, monitoring quality, and managing prompt and model changes.
- [Prompt injection](https://methodcrew.com/glossary/prompt-injection): Prompt injection is an attack where text given to a language model, typed by a user or hidden in a document, overrides the instructions the system was built with.

## Next step

Request a 30-minute scoping call: https://methodcrew.com/contact. Or email contact@methodcrew.com.