How SentinelOne evaluates the AI agents behind a 99.7% agreement rate

#The challenge: A correct verdict can hide a broken investigation

An investigation agent reviews an alert, queries the SIEM, gathers evidence, performs enrichment, and returns a comprehensive report and verdict. Score only the verdict and a broken agent passes: it can match the expert's conclusion while the evidence behind it is wrong. SentinelOne's Building agents backwards from evaluation gives the example of an agent that filtered on the wrong device identifier, pulled unrelated events, and still landed on the right answer. The output looks clean and nothing in it flags the mistake.

"There are lots of failure modes for agents. Many of them are not intuitive, and many of them cannot be captured by ordinary tests."

Tom Kuson, Harness Engineering Lead at SentinelOne

Spiraling is another failure mode. The agent runs the same query again and again with small variations, getting to the right answer only by exhausting every option. Tom's team watches for it because each extra loop costs tokens and adds minutes to the run, while the final output still looks correct.

Before Pydantic AI, the team instrumented its own agent framework with OpenTelemetry and read results out of ordinary logs and raw Python dataclasses. Tim Ng, who leads evals, saved traces and reports to local files and analyzed them after the fact. That held up for one engineer studying one experiment. It did not hold up once the team was running dozens at a time.

#The solution: Pydantic AI for the harness, Pydantic Evals and Pydantic Logfire for the proof

The team built its own agent framework in early 2023 because alternatives at the time did not fit. LangChain had started to take shape, but it was built around chains that passed the output of one prompt into the next in a fixed sequence, and Purple AI needed graph-shaped workflows closer to a finite state machine. Building their own kept the fundamentals in their hands.

SentinelOne had been building on Pydantic Validation since the team formed, down to Pydantic Settings for configuration. Typed, validated data structures were already how the team modeled the work. Pydantic AI extended that same discipline to agents, and it read as idiomatic Python rather than a framework that would reshape the codebase around itself.

"We needed a good-taste way to move quickly, and the Pydantic solution had that."

Tom Kuson, Harness Engineering Lead at SentinelOne

What decided it was what came with Pydantic AI. The team had been designing its own eval harness, choosing a framework to sit behind it, and working out how to visualize results. Pydantic Evals and Pydantic Logfire covered all three, and they connect. Pydantic Evals writes its runs as OpenTelemetry spans and Logfire reads them, so a failing score and the trace that produced it are one click apart. An engineer sees a competency drop, opens the run behind it, and reads the actual queries the agent wrote, all without stitching multiple tools together.

Logfire is built on OpenTelemetry and works with any OTel-emitting stack, which is how SentinelOne's homegrown framework was already instrumented well before Pydantic AI entered the picture.

#Pydantic Evals in production: Scoring each capability an investigation requires

A competency is one capability the work requires, scored on its own. SentinelOne evaluates an investigation the way a reviewer would examine the work itself: not on whether the final answer sounds right, but on whether each step that produced it holds up.

Tim's team started with the hardest case, a full SOC investigation, and built scenarios around it using rubrics in Pydantic Evals.

Decomposition is what made the scores useful. The team pulled apart what an investigation requires and wrote a competency for each part: forming a hypothesis worth testing, writing a correct query against a given data source, performing enrichment, following policy, and knowing when the evidence is enough to stop. Each one gets its own set of rubrics and its own suite of cases, so a failing score names the capability that broke.

Along with their custom trace evaluators, LLM-as-a-judge earned its place once the team scoped it tightly. A judge pointed at a whole investigation produces a vague verdict and can't reliably grade performance. Narrow it to a single objective, with a multitude of judges for each capability, and they produce scores the team can use to quantify capability performance.

At SentinelOne today, hundreds of evaluators gate quality and characterize performance, and the whole suite runs in CI/CD. Nothing reaches a customer environment without clearing it. As the team puts it, "in security response, unmeasurable capability is unshippable capability."

Competency scores also make model comparison concrete. Testing one frontier model against another produces a profile instead of a single number, which shows exactly where each model is strong. Some models chase context far more eagerly than others, which the team sees in tool call counts, and each competency score sits alongside what that run cost and how long it took.

"We were able to use the rubrics in very focused ways to begin to quantize that kind of performance. It's almost painfully clear where they excel versus where other models don't."

Tim Ng, Eval Engineering Lead at SentinelOne

For choosing between candidates, the Purple AI team plots each one on a radar chart, with multiple experiments running at any given time. The chart is how the team decides which candidate earns the next round of A/B testing in the product and which one to drop before spending more on it.

The multi-dimensional approach the team takes to compare and select production solutions.

#Pydantic Logfire: Shared traces, dashboards, and live agents

Logfire ended the local-files era. Tim stopped piping traces into his own storage, and results became something the team could work on together.

"It was extremely useful to log all of these traces into a collective platform for sharing. I can see what other people are working on and what experiments people are running. People get siloed pretty easily, but if you start looking through it all, you can connect a lot of experimentation data."

Tim Ng, Eval Engineering Lead at SentinelOne

That visibility changes what the team catches. When an agent spirals on a tool call, Tim isolates those traces and sends them to the team, so the fix comes out of a group looking at the same evidence instead of one engineer reconstructing it alone.

The Logfire MCP server does the first pass over results. Tim asks it to build a dashboard summarizing an experiment, which quickly turns a question about thousands of runs into a chart, then opens the individual traces in Logfire when a number looks wrong.

A dashboard generated through the Logfire MCP server to summarize an experiment before the team dives deeper into the traces.

Roughly three quarters of Tim's time goes to offline evals, and the rest to live monitoring. The Agents view in Logfire aggregates every agent running in production onto one page, with call volume, cost, and latency for each, so an agent behaving differently from the rest stands out without anyone going looking for it. That page surfaced a problem in a new agent the team had started using internally, and pointed them straight at why it was getting called so often and costing more than expected.

"Logfire helped facilitate that turnkey model lifecycle, all the way through from experiment to traceability where the agents operate. We can see exactly what the agents are doing at any time, and then go all the way to a deployment if it passes the gates."

Samuel Matos, AI Architect at SentinelOne

#Self-hosted AI observability: Kubernetes, Helm, and a security review

SentinelOne treats protection of customer data as a design requirement. Every technology that touches that data goes through a security assessment covering where it runs, what it processes, and the risk that carries. Agent traces carry that data, so Logfire went through the same review. Working with their infosec team, the Purple AI team moved a cloud proof of concept to a self-hosted deployment on their own infrastructure.

Self-hosting meant the team owned the scaling as well. Purple AI's infra engineers size the deployment to their own load using the Helm chart Logfire ships.

"The chart for deployment has been golden. It allows us to customize the deployment to the scales that we need. It's been a breeze to do it in Kubernetes for the team."

Samuel Matos, AI Architect at SentinelOne

#The results

A 99.7% agreement rate, held through constant change. Agentic Investigations runs at a 99.7% agreement rate across hundreds of thousands of investigations. Holding a number like that steady while models, prompts and tools change underneath is what the eval suite is for: every capability is scored before a change ships, so a regression surfaces in CI/CD rather than in a customer's environment.

New AI features ship 30% faster. Time to ship and evaluate a new AI feature dropped by 30% once evals and Logfire ran inside CI/CD. In a market where both attacker technique and frontier model capability change monthly, that pace is what keeps the product current.

332,000 analyst hours back. SentinelOne estimates the work has saved customers 332,000 analyst hours, time that goes back into harder incidents and proactive threat hunting.

Model choices rest on measured evidence. Competency scores decide which models graduate to A/B testing in the product, and the radar view makes the trade-offs between models legible at a glance. Logfire also makes it easy to identify and correct hot spots in specialized agent capabilities.

Deployment fits the security model. Self-hosting on Kubernetes keeps sensitive customer data inside SentinelOne's own environment, with a Helm chart the team tunes to its own scale.

#Key takeaways

• Score the capabilities, not just the answer. Break the work into the capabilities it requires, write a rubric for each, and a failing run tells you which capability broke instead of handing you one number.
• A correct answer is not a passing result. An agent can reach the right verdict from the wrong evidence. Score the path, not only the endpoint.
• Put the eval suite in CI/CD. Gating on evals is what turns evaluation from a research exercise into a shipping requirement.
• Speed is only an advantage with accuracy behind it. The faster an agent acts, the more a hidden reasoning error costs. Measuring the path is what makes the speed safe to use.
• Eval results belong where the team can see them. Traces and reports on one engineer's laptop are invisible to everyone else, and shared results connect experiments that would otherwise stay siloed.

Interested in evaluating and observing agents in production? Explore the docs or get started with Logfire.