AI Agent Monitoring, Support and Maintenance

Trace every run: the prompt, each tool call, what came back and the final answer. Then track task completion, escalation rate, latency per step, error rate, cost per conversation and test set score over time. Alert when a number leaves its normal range, and read real conversations regularly, because some failures never show up in a metric.

Yes. You cannot see inside the model, but you can record everything around it: the input, the documents it looked up, each tool call and its result, the output, how long each step took and the tokens used. That record, called a trace, is enough to find most failures. Evaluations and human review then show whether the answers were right.

Observability means you can work out why an agent did something, not just that it did it. It combines traces of each step, prompts, retrieved text, tool calls, tokens, latency, errors and quality scores. Monitoring tells you a number changed. Observability gives you the detail to find the cause, such as a tool that returned nothing.

There is no official standard, and vendors list different pillars. A practical five: traces of every step and tool call; quality, measured with evaluations and human review; cost per conversation or task; performance, meaning latency and errors; and safety, covering prompt injection and sensitive data in logs. Together they explain most production problems.

The user's input, the prompt version, the model ID, retrieved documents, each tool call with its arguments and response, the final output, token counts, latency per step and any errors, plus the conversation ID and the outcome, such as resolved or handed off. Without the prompt version and model ID, you cannot tell whether a change caused a problem.

It depends on your stack. LangSmith suits teams on LangChain or LangGraph, Langfuse suits teams that want open source and self-hosting, Arize Phoenix suits teams standardizing on OpenTelemetry, Braintrust suits teams that evaluate every release, and Datadog suits teams already on Datadog. The tool matters less than whether someone reads its alerts and acts on them.

Yes. Datadog LLM Observability, which its docs now call Agent Observability, traces LLM calls, workflows and agent runs. It has dashboards for cost, latency and token usage, quality evaluations, and scanning for sensitive data and prompt injection, and it links agent traces to the rest of your Datadog data. It is not available on Datadog's US government sites.

Yes. Langfuse is open source under the MIT license, with enterprise features licensed separately, and runs self-hosted or in its cloud. Arize Phoenix publishes its source code under the Elastic License 2.0 and you host it yourself. Open source removes the subscription, not the work: you still run the servers, handle upgrades and decide who reads the traces.

Langfuse is used to trace, evaluate and debug LLM applications and agents. Teams use it to see each step of a run, version prompts, collect user feedback, label outputs by hand and score answers with LLM-as-a-judge evaluators, where an AI model grades answers against datasets. Because it is open source and can be self-hosted, it suits teams whose logs contain customer data.

Not always. A tool shows the problem: a failed tool call, a cost spike, a falling score. It does not edit the prompt, fix the connector, rerun the tests or plan a model migration. If engineers who know the agent have time to act on alerts, a tool may be enough. If nobody owns the agent, the alerts pile up unread.

Model drift is when an AI system gets worse over time although nobody changed its code. For agents it has three common causes: customers ask different things, the rules and documents behind correct answers change, or the model behaves differently after a provider update. It is usually gradual, so it needs scheduled checks rather than waiting for complaints.

Researchers Lingjiao Chen, Matei Zaharia and James Zou tested GPT-4 in March and June 2023 on the same prime-number questions, and its accuracy fell from 84% to 51%. A business example is quieter: a support agent keeps quoting the old return window after a policy change because the knowledge base sync failed. Nothing errors. Customers just get wrong answers.

Concept drift is when the right answer to the same question changes. A customer asks a familiar question, but your return policy, prices or eligibility rules changed, so last month's correct answer is now wrong. The model did not change at all. The fix is keeping the knowledge base and business rules current, then retesting the affected questions.

Model drift is the broad term for results sliding over time. Concept drift is one cause of it: the link between a question and its correct answer changes, such as after a policy update. For LLM agents, name a second cause separately: the model's own behavior changing when a provider ships an update or you switch versions.

Data drift means the inputs changed: customers ask about a new product or use new terms, while the correct answers stay the same. The agent now sees questions it was never tested on. Model drift is the drop in quality that can follow. Catch data drift by grouping incoming questions by topic and flagging new or growing ones.

Compare against a baseline on a schedule. Rerun the same test set and compare scores with the last release. Group incoming questions by topic and watch for new ones. Track how often document search finds nothing relevant, along with escalation rate and user feedback. When one of these moves, read the conversations behind it before changing anything.

You cannot prevent all of it, but you can catch it early. Pin exact model versions instead of aliases that update. Test the replacement before a model retires. Keep the knowledge base synced and retest answers tied to changed documents. Add every real failure to the test set, and rerun it before each change and on a regular schedule.

No. A hallucination is one answer that states something false, and it can happen on day one. Drift is a trend: the share of wrong or weak answers rises over time. They are linked, because drift often shows up as more hallucinations, such as when document search stops finding the right page. You catch hallucinations per answer and drift across weeks.

It depends on seven things: model tokens, conversation or task volume, connected systems, evaluation cadence, your monitoring tool, compliance needs and required response times. We do not publish prices, because an agent handling a few internal requests and one handling thousands of customer conversations need very different support. We quote after an audit of your agent.

Running cost is mostly model usage. OpenAI and Anthropic bill per million tokens, price input and output separately, and charge more for output and less for cached input. Cost grows with volume, prompt length, retrieved text and steps per task. Add your monitoring tool and the people who review and fix the agent. Our AI agent cost guide has a worked example.

Add up four parts: model usage for the month, your monitoring tool plan, the engineering hours to review, test and fix the agent, and compliance work such as HIPAA reviews. The engineering hours often decide the total, and they depend on how often the agent changes and how fast you need responses. That is why our plans are built around hours.

It can be, if maintenance is in the business case from the start. An agent doing a high-volume, repetitive job with clear rules usually covers its upkeep. One that runs a few times a week, touches risky systems and changes constantly may not. Before building, weigh monthly review and fix time against the expected savings, and name an owner.

Usually one of four: your own engineers, the original developer on a retainer, a managed support partner, or nobody, with only a monitoring tool. Vendors such as LangSmith, Langfuse, Arize, Braintrust and Datadog provide software, not fixes. FactoryJet offers managed support plans for agents we built and agents built by others, so we are one of the options, not a neutral judge.

It depends on the plan and the severity. First response times for critical issues: Monitor within 4 hours, 7 days a week; Improve within 2 hours, 7 days a week; Operate within 1 hour, 24/7. For standard requests: Monitor within 1 business day; Improve within 8 business hours; Operate within 4 business hours. Business hours are 9am to 6pm, Monday to Friday, in your US time zone. A first response means an engineer has reviewed the issue and replied with next steps. It is not a fix time.

Monitor includes 8 engineering hours a month. Improve includes 20 engineering hours a month. Operate includes 40 or more engineering hours a month. The hours cover log review, evaluation runs, fixes, integration upkeep and the monthly report. The right plan depends on how often your agent changes, how many systems it touches and how fast you need responses. We recommend one after an audit, and plans are quoted rather than priced on this page.

A critical issue means the agent is down, an integration it depends on is failing, or it gives wrong or unsafe answers at scale. Examples: the help desk rejects every reply the agent posts, or the agent quotes an outdated policy to every customer after a failed sync. Critical issues get the fastest first response on every plan.

Requests to a retired model fail, so the agent stops working on that date. OpenAI's stated minimum notice is 6 months for generally available models, and Anthropic's is 60 days for publicly released models. We log each model ID with its retirement date, test the recommended replacement on your test set, fix prompt or parameter differences and switch early.

Yes. We start with an audit: we read the code, prompts and logs, list every integration and model version, and check for a test set. If none exists, we build one first, because nothing can be changed safely without it. Then we recommend a plan. If the agent needs structural work before it can be supported, we say so up front.