SLO monitoring in Logfire

#TL; DR

• SLI: the thing you measure. For HTTP services, usually the fraction of requests that did not return 5xx.
• SLO: the target for that measurement, e.g. 99.9% non-5xx responses over 30 days.
• Error budget: what the SLO allows you to spend. At 99.9% over 30 days, that is 43 minutes of full downtime.
• Burn rate: how fast you are spending the budget, expressed as a multiple of the sustainable rate.
• Multi-window, multi-burn-rate: fire a page when a fast window AND a slow window both show a high burn rate. Detect outages quickly, ignore short blips.
• In Logfire: burn rate is a single arithmetic expression in SQL. The same expression powers both the dashboard and the alerts.

#What an SLO actually measures

An SLO is built from three things, in order:

1. Service Level Indicator (SLI): a ratio of good events to total events. For an HTTP API, the most common indicator is non-5xx responses / total responses. For a queue, it might be messages delivered within deadline / total messages. For a database, successful queries / total queries.
2. Service Level Objective (SLO): a target for that ratio over a rolling window. "99.9% of requests over the last 30 days returned a non-5xx response."
3. Error budget: the allowed shortfall. If the SLO is 99.9%, the budget is 0.1% of requests. Over 30 days, that translates to roughly 43 minutes if the service is consistently down, or many more minutes if the failures are spread out.

The error budget is the part teams underuse. It exists to make reliability a tradeoff rather than an absolute, which means using it: when there is room, that's the time to ship the migration you have been putting off, and when it is exhausted, freeze releases until the service recovers.

The SLI we use throughout this post is HTTP availability:

availability = count(http_response_status_code < 500) / count(*)

Everything that follows generalises to any ratio SLI. Latency SLOs work the same way once you pick a threshold (for example, "good = response under 300ms") and count the responses below it.

#Picking a good SLI

The mechanics that follow are only as useful as the indicator they are wrapped around. A bad SLI produces noise no matter how careful the alerting is, so it pays to settle this before building anything on top.

A few rules that tend to hold:

• Measure what the user sees. A queue that drains successfully but takes 10 minutes is not "available" from the user's perspective. Latency belongs in the SLI, not in a separate dashboard nobody opens. When you do mix in latency, scope the SLI to incoming requests rather than every span in your trace data, so internal work does not inflate the denominator.
• Exclude what you cannot control. 4xx responses are usually client error and should not count against availability. Same for traffic from synthetic monitoring, if it is significant.
• Pick a target you can actually defend. 99.99% sounds impressive and is almost never the right answer. The error budget at 99.99% over 30 days is 4 minutes. One bad deploy and the budget is gone.
• One SLO per critical user journey. "Login works", "search returns results", "checkout completes". Aggregate availability across an entire product is hard to act on.

No single SLO covers an entire system. Set one up per service that has its own on-call rotation, and tune the goal per service based on what users actually need.

#Why a single threshold alert is not enough

The first alert most teams write looks something like this:

If the 5xx rate is above 0.1% for 5 minutes, page someone.

This works for the dramatic case where the service is hard down. It fails in two more common cases.

False positives. A brief blip, ten failed requests in a quiet minute, can easily cross 0.1% and page someone at 3am. The error rate genuinely spiked, but the underlying problem was already gone by the time the page arrived.

False negatives. A slow, sustained leak, 0.05% of requests failing for two weeks, never triggers the threshold. It silently consumes the entire monthly error budget. By the time anyone notices, there is no budget left to absorb the next real incident.

The deeper problem is that the threshold has no concept of the budget. It treats every minute the same, regardless of whether you have spent 5% or 95% of the budget for the month. SLO alerting fixes this by alerting on the rate at which the budget is being consumed.

#Burn rate, explained

The burn rate is the multiple of the sustainable error rate that you are currently experiencing.

burn_rate = observed_error_rate / (1 - SLO_goal)

For an SLO of 99.9%, the sustainable error rate is 0.1%. If the current error rate is 1.44%, the burn rate is 14.4x. At that rate, you would consume an entire 30-day error budget in roughly two days.

The same relationship rearranges into a threshold on the error rate, which is handy when configuring alerts against a metrics system that reports rates rather than ratios:

error_rate_threshold = burn_rate × (1 - SLO_goal)

For a 14.4x burn rate against a 99.9% SLO, the equivalent error rate is 14.4 × 0.001 = 1.44%. Both forms describe the same condition; pick whichever your query language makes easier to read.

Two reference points worth memorising:

• 14.4x burn rate sustained for 1 hour consumes 2% of a 30-day budget.
• 6x burn rate sustained for 6 hours consumes 5% of a 30-day budget.

These are the two thresholds from the Google SRE workbook, and we will use them as reference lines on the dashboard.

A burn rate of 1 means you are spending the budget exactly at the rate the SLO allows. Anything above 1 means the budget is shrinking faster than the rolling window can replenish it. Anything below 1 means it is recovering.

#Multi-window, multi-burn-rate alerts

A burn rate alert with a single window has a tradeoff between detection time and noise:

• Short window (5 minutes): detects incidents fast, but a single bad minute can trip the alert.
• Long window (1 hour): only fires on sustained problems, but pages arrive 30 to 45 minutes after the incident starts.

The multi-window, multi-burn-rate (MWMBR) pattern resolves this by requiring both a long window and a short window to be over the threshold at the same time. The long window provides confidence (the issue is real and sustained), and the short window provides recency (the issue is still happening right now, not an hour ago that has since recovered).

The standard configuration from the SRE workbook uses two alert tiers:

The first two tiers catch acute incidents. The third catches the slow leak that would otherwise drain the budget unnoticed.

#Building the dashboard

A Logfire dashboard is a collection of panels, each backed by a SQL query against the records table (the table of spans captured from your services). For an SLO dashboard, three variables make the whole thing reusable:

• service_name: the set of services to include.
• slo_goal: the SLO target as a decimal, default 0.999.
• resolution: the time bucket for the time-series panels, default 5 minutes.

The panels below are written against those variables and the records table. Drop them into a new dashboard one at a time.

SELECT
  ROUND(
    100.0 * COUNT(*) FILTER (WHERE http_response_status_code < 500)
         / NULLIF(COUNT(*), 0),
    4
  ) AS "Availability %"
FROM records
WHERE array_has($service_name, service_name)
  AND http_response_status_code IS NOT NULL

SELECT
  ROUND(
    GREATEST(
      0,
      100.0 * (
        1.0 - (COUNT(*) FILTER (WHERE http_response_status_code >= 500)::double
               / NULLIF(COUNT(*), 0))
              / (1.0 - CAST($slo_goal AS DOUBLE))
      )
    ),
    2
  ) AS "Budget Remaining %"
FROM records
WHERE array_has($service_name, service_name)
  AND http_response_status_code IS NOT NULL

SELECT
  time_bucket($resolution, start_timestamp) AS x,
  ROUND(
    (COUNT(*) FILTER (WHERE http_response_status_code >= 500)::double
     / NULLIF(COUNT(*), 0))
    / (1.0 - CAST($slo_goal AS DOUBLE)),
    2
  ) AS burn_rate,
  14.4 AS fast_burn_threshold,
  6.0  AS slow_burn_threshold
FROM records
WHERE array_has($service_name, service_name)
  AND http_response_status_code IS NOT NULL
GROUP BY x
ORDER BY x

SELECT
  service_name,
  COUNT(*) AS total_requests,
  COUNT(*) FILTER (WHERE http_response_status_code >= 500) AS errors_5xx,
  ROUND(
    100.0 * COUNT(*) FILTER (WHERE http_response_status_code < 500)
         / NULLIF(COUNT(*), 0),
    4
  ) AS "availability_%",
  ROUND(
    (COUNT(*) FILTER (WHERE http_response_status_code >= 500)::double
     / NULLIF(COUNT(*), 0))
    / (1.0 - CAST($slo_goal AS DOUBLE)),
    2
  ) AS burn_rate
FROM records
WHERE array_has($service_name, service_name)
  AND http_response_status_code IS NOT NULL
GROUP BY service_name
ORDER BY burn_rate DESC
LIMIT 20

#Configuring the alerts

A Logfire alert is a SQL query that runs on a schedule (every minute, every five minutes, and so on). When the query returns rows, the alert fires and posts to the notification channel attached to it. When it returns zero rows, the alert is silent.

That zero-or-more-rows convention shapes how MWMBR alerts are written. The query must compute the burn rate over the long window and the short window, grouped by service, and return one row per service that is burning above the threshold in both windows. The row identifies the offending service directly in the alert payload, so on-call does not have to drill into the dashboard to find out which service triggered the page.

WITH long_window AS (
  SELECT
    service_name,
    (COUNT(*) FILTER (WHERE http_response_status_code >= 500)::double
     / NULLIF(COUNT(*), 0))
    / 0.001 AS burn_rate
  FROM records
  WHERE start_timestamp > now() - INTERVAL '1 hour'
    AND http_response_status_code IS NOT NULL
  GROUP BY service_name
),
short_window AS (
  SELECT
    service_name,
    (COUNT(*) FILTER (WHERE http_response_status_code >= 500)::double
     / NULLIF(COUNT(*), 0))
    / 0.001 AS burn_rate
  FROM records
  WHERE start_timestamp > now() - INTERVAL '5 minutes'
    AND http_response_status_code IS NOT NULL
  GROUP BY service_name
)
SELECT
  long_window.service_name,
  ROUND(long_window.burn_rate, 2)  AS long_burn,
  ROUND(short_window.burn_rate, 2) AS short_burn
FROM long_window
JOIN short_window USING (service_name)
WHERE long_window.burn_rate  > 14.4
  AND short_window.burn_rate > 14.4

WITH long_window AS (
  SELECT
    service_name,
    (COUNT(*) FILTER (WHERE http_response_status_code >= 500)::double
     / NULLIF(COUNT(*), 0))
    / 0.001 AS burn_rate
  FROM records
  WHERE start_timestamp > now() - INTERVAL '6 hours'
    AND http_response_status_code IS NOT NULL
  GROUP BY service_name
),
short_window AS (
  SELECT
    service_name,
    (COUNT(*) FILTER (WHERE http_response_status_code >= 500)::double
     / NULLIF(COUNT(*), 0))
    / 0.001 AS burn_rate
  FROM records
  WHERE start_timestamp > now() - INTERVAL '30 minutes'
    AND http_response_status_code IS NOT NULL
  GROUP BY service_name
)
SELECT
  long_window.service_name,
  ROUND(long_window.burn_rate, 2)  AS long_burn,
  ROUND(short_window.burn_rate, 2) AS short_burn
FROM long_window
JOIN short_window USING (service_name)
WHERE long_window.burn_rate  > 6
  AND short_window.burn_rate > 6

WITH long_window AS (
  SELECT
    service_name,
    (COUNT(*) FILTER (WHERE http_response_status_code >= 500)::double
     / NULLIF(COUNT(*), 0))
    / 0.001 AS burn_rate
  FROM records
  WHERE start_timestamp > now() - INTERVAL '3 days'
    AND http_response_status_code IS NOT NULL
  GROUP BY service_name
),
short_window AS (
  SELECT
    service_name,
    (COUNT(*) FILTER (WHERE http_response_status_code >= 500)::double
     / NULLIF(COUNT(*), 0))
    / 0.001 AS burn_rate
  FROM records
  WHERE start_timestamp > now() - INTERVAL '6 hours'
    AND http_response_status_code IS NOT NULL
  GROUP BY service_name
)
SELECT
  long_window.service_name,
  ROUND(long_window.burn_rate, 2)  AS long_burn,
  ROUND(short_window.burn_rate, 2) AS short_burn
FROM long_window
JOIN short_window USING (service_name)
WHERE long_window.burn_rate  > 1
  AND short_window.burn_rate > 1

#Backtesting the alerts against past incidents

Before sending these alerts to PagerDuty, run them against history. The cleanest place to do this is the SQL Explorer in Logfire, where you set the time range from the time picker in the UI instead of writing it into the SQL.

Pick a past incident, set the Explorer time range to a one-hour window ending at the moment the alert would have evaluated, and run the long-window query without any time filter:

SELECT
  service_name,
  COUNT(*) AS total,
  COUNT(*) FILTER (WHERE http_response_status_code >= 500) AS errors_5xx,
  ROUND(
    (COUNT(*) FILTER (WHERE http_response_status_code >= 500)::double
     / NULLIF(COUNT(*), 0))
    / 0.001,
    2
  ) AS burn_rate
FROM records
WHERE http_response_status_code IS NOT NULL
GROUP BY service_name
ORDER BY burn_rate DESC

The UI time range becomes the long window, so the rows that come back are exactly what the long-window CTE in the alert would have produced at that moment. Compare each burn_rate against the corresponding threshold (14.4 for the 1-hour tier, 6 for the 6-hour, 1 for the 3-day).

To check the short window, shrink the time picker to the matching span (5 minutes, 30 minutes, or 6 hours) and re-run the same query. The alert fires only when the burn rate exceeds the threshold in both ranges, so the backtest succeeds when both queries return the offending service above the threshold and quietly when at least one does not.

This is also a useful way to argue about thresholds with the rest of the team. Rather than picking numbers from the workbook and hoping, you can scrub the time picker across the last month of production traffic and show what each setting would have caught and what it would have ignored.

#FAQ

SELECT
  ROUND(
    (COUNT(*) FILTER (WHERE duration >= 0.3)::double
     / NULLIF(COUNT(*), 0))
    / (1.0 - CAST($slo_goal AS DOUBLE)),
    2
  ) AS burn_rate
FROM records
WHERE array_has($service_name, service_name)
  AND span_name = 'GET /api/search'
  AND duration IS NOT NULL

#Get started

Logfire's free tier includes 10M spans per month, enough to back an SLO dashboard for a real service.

1. Sign up: logfire.pydantic.dev
2. Instrument an HTTP service: any service that records http_response_status_code on its spans will work with the queries above. The Logfire docs cover the standard OpenTelemetry instrumentations.
3. Build the dashboard: create a new dashboard, add the service_name, slo_goal, and resolution variables, and paste the panel queries one at a time.
4. Configure the alerts: start with the 1-hour page-level alert and add the others once you have seen it fire (or not) for a week.
5. Questions or feedback: reach out on Slack.

The dashboard is mostly for retrospectives and weekly reviews. The alerts are what on-call actually sees at 3am, so they are where tuning effort pays back the fastest.