Skip to content
Pydantic Docs

Airflow

See every task your Airflow pipelines run (how long each took and how it ended), plus Airflow’s own metrics, in Logfire. Each task run becomes a span (one unit of work with a name, a start, and a duration), and related spans link into a trace (the full journey of one run), so you can follow a run across your pipeline.

Airflow has native OpenTelemetry support for traces and metrics: it builds an internal exporter (the piece that sends telemetry out) and ships data straight to the backend you point it at. So instead of a logfire.instrument_* call, you turn Airflow’s own support on and aim it at Logfire with a couple of settings.

What you’ll capture

  • Each task run as a span, with its duration and outcome
  • The full run of a pipeline as a trace
  • Airflow’s built-in metrics

Before you start

You’ll need a Logfire project. Open Add data in your project (top navigation) and follow the setup for your language: it signs your machine in with logfire auth (a browser sign-in, no token to copy) and, for production or other languages, creates a write token (the credential your app uses to send data). New to Logfire? Start with Getting Started.

Installation

Logfire has no Airflow-specific extra, and you don’t install logfire in your Airflow environment: Airflow exports OpenTelemetry to Logfire directly. You do need Airflow’s own otel extra, which pulls in the OpenTelemetry exporter Airflow uses:

Terminal
pip install 'apache-airflow[otel]'

Everything else below is set through environment variables and airflow.cfg.

Usage

Two steps: pass your write token as an OpenTelemetry header, then point Airflow’s otel_* settings at Logfire.

First, set the header so Airflow authenticates to Logfire. Run this in the same shell (or set it in the environment) where Airflow runs, with LOGFIRE_TOKEN holding your write token:

Terminal
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=${LOGFIRE_TOKEN}"

Then turn on OpenTelemetry in your airflow.cfg and aim it at Logfire’s endpoint:

airflow.cfg
[metrics]
otel_on = True
; use logfire-eu.pydantic.dev for the EU region
otel_host = logfire-us.pydantic.dev
otel_port = 443
otel_prefix = airflow
otel_interval_milliseconds = 30000  # The interval between exports, defaults to 60000
otel_ssl_active = True

[traces]
otel_on = True
; use logfire-eu.pydantic.dev for the EU region
otel_host = logfire-us.pydantic.dev
otel_port = 443
otel_prefix = airflow
otel_ssl_active = True
otel_task_log_event = True

For the full list of settings, see Airflow’s traces and metrics documentation.

Verify it worked

Trigger a pipeline run, then open the Live view. Within a few seconds you’ll see spans for the task runs: click one to see its duration and outcome. Airflow’s metrics appear in your project’s dashboards.

Troubleshooting

Not seeing your task runs? Check that OTEL_EXPORTER_OTLP_HEADERS is set in the environment Airflow actually runs in, that LOGFIRE_TOKEN holds a valid write token, that otel_on = True under both [traces] and [metrics], and that otel_host matches your region (logfire-us.pydantic.dev or logfire-eu.pydantic.dev). On apache-airflow older than 2.10.4, use the Collector route below.

Advanced

Older Airflow: via an OpenTelemetry Collector

If your apache-airflow is older than 2.10.4, you can’t set the OTEL_EXPORTER_OTLP_HEADERS environment variable, so Airflow can’t authenticate to Logfire directly. Instead, run an OpenTelemetry Collector (a standalone agent that receives telemetry from your apps and forwards it on) in front of Logfire. Airflow sends to the collector; the collector adds your token and forwards to Logfire.

Follow the OpenTelemetry Collector installation guide to set one up, then use this configuration:

otel-collector-config.yaml
receivers:  # (1)
otlp:
  protocols:
    grpc:
      endpoint: "0.0.0.0:4317"
    http:
      endpoint: "0.0.0.0:4318"

exporters:  # (2)
debug:  # (3)
otlphttp:
  endpoint: https://logfire-us.pydantic.dev/  # or https://logfire-eu.pydantic.dev/ for the EU region
  compression: gzip
  headers:
    Authorization: "Bearer ${env:LOGFIRE_TOKEN}"  # (4)

processors:
batch:
  timeout: 1s
  send_batch_size: 32768

extensions:
health_check:
  endpoint: "0.0.0.0:13133"

service:  # (5)
extensions: [health_check]
pipelines:
  traces:
    receivers: [otlp]
    processors: [batch]
    exporters: [debug, otlphttp]
  metrics:
    receivers: [otlp]
    processors: [batch]
    exporters: [debug, otlphttp]

Receivers collect data from your applications.

See the OpenTelemetry Collector Receiver section for more.

  1. Exporters send data on. The otlphttp exporter sends it to Logfire.

  2. The debug exporter prints what’s being sent to the console: useful while setting things up, and safe to remove in production.

  3. The Authorization header carries your Logfire write token. ${env:LOGFIRE_TOKEN} is replaced by the environment variable of the same name.

  4. The service section wires up the pipelines: traces for trace data, metrics for metrics.

Then point Airflow at the collector (running locally, no TLS) in your airflow.cfg:

airflow.cfg
[metrics]
otel_on = True
otel_host = localhost
otel_port = 4318
otel_prefix = airflow
otel_interval_milliseconds = 30000  # The interval between exports, defaults to 60000
otel_ssl_active = False

[traces]
otel_on = True
otel_host = localhost
otel_port = 4318
otel_prefix = airflow
otel_ssl_active = False
otel_task_log_event = True

Reference