Skip to main content

Optimize

Prompt optimization from production runs

Version every prompt, let the optimizer read live runs and propose changes with evidence. No eval suite required to start.

Start optimizing free Read the docs
The Logfire prompt editor for the acme organization, open on a prompt named refund_policy at version 3. The template reads 'You are a refund agent for Acme', references the managed variables refund_safety_rules and refund_auto_approve_limit, and interpolates customer_name, tier, order_id and amount. A Detected syntax panel lists both managed variable references, and the version control offers Draft (Based on v3) beside a Save as v4 action. The right half previews one run against the refund_disputes dataset on the over_limit_vip scenario: a system input, a user input asking for a full $412 refund, a model picker on openai / gpt-5-mini with a Run button, and the assistant's reply opening a case rather than approving above the limit.

Why runtime control

A prompt change has no compiler to catch it

A prompt change is a production change with no type checker and no failing test. The only way to know a rewrite helped is evidence: the live runs it would have improved, the scores it moves, the reviewer who confirms it. That takes managed prompts so versions are real objects, an optimizer that reads your production runs, and human judgment where it counts.

Pydantic Logfire has all of it next to the telemetry it explains, so the loop closes.

How it works

One continuous loop, from failure to fix

  1. Fail A bad run is a trace in Logfire, one of the failures scattered across your traffic.
  2. Propose The optimizer reads production runs and proposes a prompt edit. When the fault isn't the prompt, it names the real cause.
  3. Confirm Accept the diff or gate promotion on an eval when you want the change measured against a baseline.
  4. Roll out Move a label to a new version, canary it on a slice of traffic before rolling out to everyone.
The optimization loop, drawn as a cycle. Production traffic runs left to right, a failure drops out of it, and the cycle runs Fail, a trace in production, then Propose, an edit and cited runs, then Confirm, a verdict or an eval, then Roll out, a label on a version, and back into traffic.

What you get

Find it, ship it

  • A prompt named refund-policy at version 2. The template greets the customer by name, keeps replies short, and carries a conditional block that escalates immediately when the customer is VIP. Template parameters customer_name and customer.vip sit under it, beside a Draft (Based on v2) version picker and a Save as v3 action.

    Prompts

    Every prompt is a versioned object with history, not a string pasted into three services. A change is a new version you can diff and roll back to.

  • The Managed Variables panel with a New variable action. Under boolean, enable_new_ui is serving latest and described as a feature flag for the new dashboard UI. Under Other, agent_config is serving latest across 2 labels and 2 versions, and holds the model, temperature and system prompt for the AI agent.

    Managed variables

    Any Pydantic model becomes a versioned variable, from a boolean flag to a whole agent config. Mark one external for web and mobile.

  • A Proposed change panel: corrected the database schema in the prompt to match the actual tables and columns. A side-by-side diff strikes out the wrong table names client_accounts, catalog_items and sales_transactions and inserts the actual customers, products and orders columns. A Why this proposal disclosure sits beside a Medium Confidence badge and a Copy details action.

    Evidence-cited proposals

    The optimizer reads recent production runs and proposes one prompt edit at a time. If a claim can't cite a run you can open, it's rejected.

  • A scenario named 'Angry customer, order delayed', with Save and exit and Delete scenario actions. Under a Messages tab, a system input carries the prompt reference, a user message reads 'I am having trouble with {{topic}}. Can you help?', and a tool call result for fetch_order follows.

    Scenarios

    A named test case with its real message sequence, template parameters and tool calls. Sweep a whole dataset, up to 500 cases per run.

  • A prompt named support-agent compared with Production, plus one line and minus three lines. The Production column at version 2 takes 90 percent of traffic and strikes out four lines of florid greeting copy; the Canary column at version 3 takes 10 percent and adds 'Keep replies short, two sentences max.'

    Rollouts

    Point production at one version and a canary at another, split 90/10. Diff them line by line, then move the label to promote or roll back.

  • An Operational metrics table comparing latency, token use and cost across two runs. Task duration goes from 765 to 810 milliseconds, input tokens from 62 to 269, output tokens from 41 to 51, and cost per run from $0.008 to $0.014, each marked worse, with input tokens worse by 334 percent.

    Measured in your traces

    Every resolution tags its spans with the label and version, so you compare latency, tokens and cost between v1 and v2.

Use cases

What people use Optimize for

  • Fix a failure you couldn't see

    The optimizer reads production runs at scale where rare failures hide, finds the pattern behind them, and offers the prompt edit with evidence.

    Read how to optimize

  • Change a prompt without shipping code

    Edit the version in Logfire, promote it with a label, roll it back if it was wrong. No pull request, no release window, no redeploy.

    Use prompts in your app

  • Roll a model change out to 5% first

    Percentage rollouts and attribute targeting, with every resolution recorded as a span, so you can query the variants like anything else.

    Read the rollouts guide

Customer stories

Teams who iterate faster than they deploy

FAQ

What is prompt optimization?

Prompt optimization is improving a prompt based on evidence from real runs rather than guesswork. In Pydantic Logfire, the optimizer reads what actually happened across many runs, finds the pattern behind the failures, and proposes a change you can measure against the current version.

How does the Logfire optimizer decide what to change?

Logfire reads recent production runs with failures weighted highest, and proposes one prompt edit at a time. Every claim in the proposal cites the runs behind it. If a proposed change can't point to a run you can open, Logfire rejects it and the optimizer tries again. You review the diff and accept it before anything about the running agent changes.

Do I need an eval suite to optimize prompts?

No. The optimizer starts from your live traces: it reads recent runs, finds the failures worth fixing, and proposes the change with the runs as evidence. An eval suite is useful once you want to gate promotion on numbers, because it turns a proposal into a measured comparison against a baseline. Tracing alone is enough to begin.

What happens when the problem isn't the prompt?

The Logfire optimizer says so instead of changing the prompt. When failures trace to a down dependency, a broken tool, or a quota, it files a finding that names the real cause and points at the runs that prove it. Findings are grouped by owner: provider availability, quota and billing, tools and integrations, runtime and infrastructure, configuration, and model limitations.

Can I roll a prompt change out gradually in Logfire?

Yes. Prompts are versioned and served by label, so a new version can take a fraction of traffic before it takes all of it. Because the version that served each request is recorded on its trace, splitting the results by version is a SQL query. A regression shows up in the scores of a slice before the whole user base sees it.

Get started with Logfire for free

Sign up today and see your first trace in under 2 minutes

01 — Install 

$ uv add logfire

Resolved 7 packages in 128ms
Installed 7 packages in 89ms
+ logfire

02 — Instrument 

import logfire

logfire.configure()
logfire.info('Hello {name}', name='world')

Free up to 10m records per month No credit card needed