LLM Observability in 2026: How to See What Your AI Apps and Agents Are Actually Doing
Last week I went back to my Codex Cloud post, the one about agents that keep working after you close the laptop. One question kept bugging me. If the agent runs while nobody is watching, how do you find out later what it did, what it cost, and whether the result was any good?
People call this LLM observability, or AI observability once agents are involved. I spend my days around network observability, so I’m used to asking “what actually happened on the wire?” This post is my plain-English map of the same question for LLM apps, as things stand in October 2026, with the tools I’d look at first.
What LLM observability actually means
Classic observability gives you logs, metrics and traces for code. LLM observability keeps all of that and adds the parts that are special to models: the prompt, the response, the tokens, the tool calls, and some judgement about quality.
I think of it as four jobs.
Tracing prompts and agent runs
A trace is the step-by-step record of one request. For a simple chatbot that might be one model call. For an agent it can be a long chain: plan, call a tool, read a file, call the model again, retry, answer. You want every step on one timeline, with inputs, outputs and timing, so you can point at the exact step that went wrong.
Evals
Evals answer “was this output good?” You can score outputs with code checks (did it return valid JSON?), with an LLM acting as a judge, with human labels, or with thumbs-up and thumbs-down from users. Run them on a fixed test set before you ship, and on a sample of live traffic after.
Cost and latency
Model bills follow tokens, so you want token counts per call, per user and per feature. On the speed side, total latency matters, but so does time to first token, because that is the wait your user actually feels in a chat window.
Drift, hallucinations and safety
Quality moves over time. A provider updates a model, your documents change, users start asking new kinds of questions. The practical way to catch drift is to keep scoring a slice of live traffic and chart those scores week by week. Hallucination checks usually live in the same place, as an eval that asks “does this answer stick to the sources we gave it?” Safety monitoring adds checks for leaked personal data and prompt injection attempts.
Why your APM dashboard isn’t enough
Traditional APM is great at telling you a request returned 200 in 900 milliseconds. For an LLM app that tells you almost nothing about the thing users care about.
A confident wrong answer looks perfectly healthy in APM. So does an agent that picked the wrong tool three times before getting lucky. So does a prompt change that quietly doubled your token use. The HTTP layer is fine in all three cases.
It reminds me of network flow data. Flows show who talked to whom and how much, which is useful, but they can’t tell you whether the conversation made sense. For that you need to see inside the payload. LLM observability is how you see inside the payload of a model call. You run it next to APM, and keep both.
OpenTelemetry is becoming the common language
The change I’d pay most attention to this year is about standards. The OpenTelemetry GenAI semantic conventions, the shared names for LLM telemetry, moved into their own repository, open-telemetry/semantic-conventions-genai, in May 2026. They cover model spans, agent spans, metrics and MCP.
In practice that means standard span names like invoke_agent and execute_tool, and standard metrics for things like time to first token and input, output, cached and reasoning tokens. The conventions are still marked “Development”, so expect some names to change.
This matters because if your app emits these attributes, you can switch backends without re-instrumenting everything. Several of the tools below accept OpenTelemetry data directly.
The tools I’d look at first
I kept this list short on purpose. Everything below is checked against each product’s own docs as they read this week. I’ve left out pricing because it changes too often to be useful in a post like this.
Langfuse
Langfuse is open source and self-hostable, and its tracing is based on OpenTelemetry. Traces include LLM calls plus retrieval, embeddings and API calls. You can group multi-turn chats into sessions, view agents as graphs, and track cost per user. It also covers prompt versioning, datasets and experiments, and evals on production traces using LLM-as-a-judge, code evaluators, user feedback or annotation queues. Alerts can go to Slack, GitHub Actions or webhooks.
Arize Phoenix
Arize Phoenix is open source too, built on OpenTelemetry and OpenInference instrumentation. It accepts traces over OTLP and has auto-instrumentation for frameworks like LangChain, LlamaIndex and DSPy. You get evals (LLM-based, code-based or human labels), a prompt playground, span replay, and datasets for experiments. I like that you can start it locally with one command and self-host on Docker or Kubernetes later.
LangSmith
LangSmith Observability from LangChain covers tracing, dashboards and alerts, online evaluations through automation rules, and feedback collection with annotation queues. It works with many frameworks beyond LangChain, including OpenAI and Anthropic. You can run it in their cloud, in a hybrid setup, or self-hosted.
Datadog LLM Observability
If your team already lives in Datadog, look here first. The docs now call it Agent Observability. A trace can be a single inference, a fixed workflow, or a dynamic agent run. It ships dashboards for cost and latency, scans for and redacts sensitive data, and flags prompt injection attempts. It can also ingest OpenTelemetry GenAI traces (version 1.37 and up) without Datadog’s own SDK or the Datadog Agent.
Helicone
Helicone takes a gateway approach. You point your OpenAI-compatible client at its gateway, and every request gets logged with cost, latency and errors. It also does provider fallbacks and session views for multi-step agent workflows. If you’d rather use your own provider keys, there’s an observability-only mode.
How I’d pick one
My rough rule of thumb looks like this.
- Want open source and self-hosting? Start with Langfuse or Phoenix.
- Already building on LangChain? LangSmith will feel the most natural.
- Already paying for Datadog? Keep LLM traces next to your APM and infra data.
- Want logging with almost no code change? A gateway like Helicone is the quickest start.
Whatever you choose, instrument with OpenTelemetry GenAI attributes where you can. That keeps your exit door open.
Cost tracking matters more as model choices multiply. When I wrote about GPT-6.1 Sol, the big pitch was a much lower token price than Astra. You can only tell whether a cheaper model is good enough for your work if you’re already logging tokens and scoring outputs.
Reading long technical documents closely is a related habit, and I looked at Google Deep Review, a tool built for exactly that.
Your first week: a small setup that pays off
You don’t need a big platform project to start. Here’s what I’d do this week:
- Trace one feature. Pick your busiest LLM flow and send traces to one tool. Langfuse or Phoenix are fine for a first try.
- Log tokens and time to first token. Break cost down by feature and by user.
- Write five evals. Use real bad answers from support tickets or chat logs as test cases. Run them on every prompt change.
- Score 1 to 5 percent of live traffic with an LLM judge, and check the chart every Monday for drift.
- Add one safety check. Start with personal data in outputs or prompt injection in inputs.
If you’re testing newer, more capable models like GPT-6 Astra in agent workflows, do step one before you scale anything. Then open three agent runs from yesterday and check you can explain every step, every token and every tool call. If you can, your setup is working.
Watching agents is half the job; the other half is limiting what they can reach, which I break down in AI agent containment and why the sandbox matters.
Leave a comment