From dashboards nobody checks to regressions that find themselves

#The challenge: observability a human reads is not a feedback loop

Every team sends telemetry somewhere. Where agents write most of the code, coverage is not the hard part. When an agent completes a change, "it merged" and "it behaves as we intended" are two different facts, and non-deterministic LLMs guarantee a gap between them. Closing that gap manually doesn’t scale: someone has to remember to look, export a dashboard, and notice a graph moved.

AutonomyAI worked through AI observability tooling before landing on Logfire. They tried other key players in the market such as Datadog’s LLM observability and Langfuse, which fell short. Then they tried Logfire; the difference was that an agent could interrogate the data itself.

#The solution: AI observability your agents can query

Pydantic Logfire is built on OpenTelemetry and exposes span data as SQL. The Logfire MCP server gives a coding agent direct query access to that data. So an AutonomyAI agent can ask a live production question, "is this code path firing, and is it doing what we said?", and get an answer from real traffic rather than inferring from the outside.

That single property is what turns observability from a read-out into a loop. AutonomyAI’s client-facing agents write to Logfire, and their internal R&D agents query it through the Logfire MCP server. In practice the team lives on the MCP surface, instead of a dashboard.

"Most of our engineers and our internal systems now use the Logfire MCP for basically everything. We barely open the Logfire UI, just because we are so happy with MCP."

— Tammuz Dubnov, co-founder and CTO at AutonomyAI

#Logfire in production: three places the loop closes

The agent debugs from the same logs an engineer would. An engineer debugging an issue reaches for the logs, and AutonomyAI’s agents get the same. While an agent is building, it queries Logfire directly to check what occurred: whether the path fired, what it returned, where it broke. An agent with that access fixes its own mistakes rather than escalating each one to a person.

Review that continues past merge. It does not stop at merge. After a change ships, agents query real Logfire traces over the following days and compare intended behavior against actual behavior. Because the traces are agent-queryable, the system catches its own regressions. Nobody has to remember to go look.

Production signal becomes prioritized work. The divergences that post-merge review finds get filed as tracked follow-ups in Linear and resurface in the work queue. Part of the team’s backlog is generated by agents reading production telemetry, not only by humans filing tickets. That is the step Tammuz calls the cornerstone of how the team now runs R&D: a plan, build, review, ship, observe, re-prioritize loop where Logfire is the "observe" step, and where being agent-queryable wires it straight back into "re-prioritize" without a human in the middle.

"A cornerstone of how I am managing R&D is Pydantic Logfire and the feedback loop it enables for our agents."

— Tammuz Dubnov, co-founder and CTO at AutonomyAI

#The results

Regressions that find themselves. Over five weeks, AutonomyAI ran 105 merged PRs through post-merge behavioral review against production telemetry. The review caught 12 changes that looked shipped but were silently not firing in production, the kind of no-op that a passing CI run and a green merge will happily hide. It surfaced 65 issues, all found by proactively querying production traces after merge rather than waiting for a bug report. 55 were genuine, and 43 became tracked tickets.

Proven at production scale. Logfire ingested over 35 million spans in three weeks, north of 2 million on a typical workday and above 3.2 million at peak. That is the live telemetry the agents query against in production.

More output, less churn. Two era metrics frame the shift, stated as correlation rather than attribution. In the last 30 days the team merged 412 PRs and opened 524, against a 2025 pre-Logfire average of about 113 merged per month, roughly 3.6x the throughput. Over the same period, the revert rate fell from 1.18% to 0.49%.

A fraction of the Datadog cost. Logfire runs about 3.5x cheaper than Datadog. Some infrastructure engineers still reach for Datadog out of habit, so the two run side by side, but the agent and LLM observability that the R&D loop depends on runs on Logfire.

The loop starts with Pydantic AI. The team weighed OpenAI’s agent framework, LangChain, and others before settling on Pydantic AI. Logfire ingests spans from any stack that emits OpenTelemetry, but Pydantic AI ships with built-in Logfire instrumentation. Every agent run, model call, and tool call arrives as a structured span, so when AutonomyAI’s agents query production, the telemetry the loop depends on is already there. "The fact that we were able to stand it up ourselves is a testament to the fact that Pydantic built something everybody can start with," says Alyosha Makarov, VP of Sales at AutonomyAI.

#Key takeaways

• Observability only closes a loop if your agents can query it, not just your humans. A dashboard a person has to remember to open is not a feedback loop.
• Most of what looks like "making the agent more capable" is really making the system observable to the agent. An agent that can observe its own runtime behavior fixes its own mistakes instead of escalating them.
• Review does not have to stop at merge. When traces are agent-queryable, regression detection closes itself; no human has to notice the graph moved.
• Let production telemetry generate backlog. Divergences that agents find in real traffic can become tracked tickets directly, so some of your prioritized work is written by the system observing itself.
• For an agent-driven team, direct query access to trace data can matter more than a polished UI. AutonomyAI barely opens the Logfire dashboard because the Logfire MCP server gives their agents everything they need in SQL.

Building agents that need to query their own production traces? Get started for free with Pydantic Logfire.