#Preface

Querying data in Logfire is about to get dramatically faster. Consider >1,000 times faster in some cases. Queries that previously timed out after 30 seconds now complete in under a second.

"Seeing much faster <customer_attribute> query times! Feels basically instant now", says an early access customer

This technical blog post details how we achieved this through dynamic shredding, an optimization for querying semi-structured attributes data. We'll begin to roll out this feature to all customers in February 2026.

#Logfire Data Model

OpenTelemetry data (the data that Logfire ingests) is semi-structured data. It contains a mix of:

• Fixed schema fields (timestamp, span name, etc.)
• Semi-structured attributes (HTTP status codes, user IDs, regions, etc.)

Semi-structured data is sent via attributes (per-span key-value pairs) or resource attributes (otel_resource_attributes in Logfire; these are per-resource key-value pairs where a resource is a service/k8s pod/etc, essentially an instance of your application or service).

When you write code like:

app = FastAPI()

@app.get('/user/{user_id}')
async def handle_request(user_id: str) -> dict[str, str]:
  with logfire.span(f'handle_request for user {user_id}', user_id=user_id):
    return {"user_id": user_id}

This generates an export along the lines of:

{
  "resource": {
    "attributes": {
      "service.name": "my-web-service",
      "region": "us-west-2"
    }
  },
  "spans": [
    // Manual span created by user code
    {
      "span_name": "handle_request for user user123",
      "start_time": "2024-01-01T12:00:00Z",
      "end_timestamp": "2024-01-01T12:00:01Z",
      "message": "handle_request for user user123",
      "trace_id": "0194c7b1e5f4a3b2c1d0e6f7a8b9c0d1",
      "span_id": "abcdef1234567890",
      "parent_span_id": "1234567890abcdef",
      "attributes": {
        "user_id": "user123",
        "http.status_code": 200,
      },
    },
    // Automatic span created by OpenTelemetry FastAPI instrumentation
    {
      "span_name": "GET /user/{user_id}",
      "start_time": "2024-01-01T12:00:00Z",
      "end_timestamp": "2024-01-01T12:00:01Z",
      "message": "GET /user/user123",
      "trace_id": "0194c7b1e5f4a3b2c1d0e6f7a8b9c0d1",
      "span_id": "1234567890abcdef",
      "parent_span_id": null,
      "attributes": {
        "http.method": "GET",
        "http.response": 200,
      },
    }
  ]
}

I'm using JSON here for illustration, although we support various wire formats (OTLP protobuf, JSON, etc.) most data is sent as protobuf over gRPC. There's also some details like timestamps being represented as integers (epoch nanos) rather than ISO strings and an extra level of nesting from scopes that I've omitted for clarity.

A couple things to note here:

1. Some fields like start_timestamp, end_timestamp, span_name, and message are fixed-schema fields that exist for every span and are strongly typed.
2. Attributes are semi-structured and can vary widely between spans. Some attributes like http.status_code and http.method are common and have semantic meaning in the OpenTelemetry spec, while others like user_id are application-specific and could be an integer in some places and a string in others.

We ultimately store this data as Parquet files in object storage, with a columnar schema roughly like:

start_timestamp: Timestamp(Microsecond, "+00:00")
end_timestamp: Timestamp(Microsecond, "+00:00")
span_name: Utf8View
message: Utf8View
trace_id: FixedSizeBinary(16)
span_id: FixedSizeBinary(8)
parent_span_id: FixedSizeBinary(8)
attributes: Utf8View (JSON)
otel_resource_attributes: Utf8View (JSON)

Some background on columnar storage, Parquet and zone map pruning is beneficial to understand the rest of this blog post. I recommend reading MotherDuck's excellent overview of the topic.

SELECT count(*) FROM records WHERE duration < 0;

SELECT count(*) FROM records WHERE _lf_attributes_http.response.status_code = 500;

SELECT count(*) FROM records WHERE attributes->'http.response.status_code' = 500;

SELECT * FROM events WHERE attributes->>'my_llm_response' like '%important%';

┌─────────────────────────────────────────┐
│  Cost 1: Download Entire _lf_attributes │
│  (could be GBs of data)                 │
└─────────────────────────────────────────┘
                    ↓
┌──────────────────────────────────────────┐
│  Cost 2: Poor Compression                │
│  Variable structure → zstd 10% efficiency│
└──────────────────────────────────────────┘
                    ↓
┌─────────────────────────────────────────┐
│  Cost 3: JSON Parsing                   │
│  Jiter does lazy parsing, worst case    │
│  must parse entire row.                 │
└─────────────────────────────────────────┘
                    ↓
┌─────────────────────────────────────────┐
│  Cost 4: In-Memory Operations           │
│  Create copies of extracted values      │
│  Do comparisons in memory               │
└─────────────────────────────────────────┘

SELECT * FROM events WHERE attributes->>'user_id' = 'user123';

#The Solution: Dynamic Shredding

Rather than storing all attributes in a single JSON column, we automatically extract the most frequently accessed attributes into separate, strongly-typed columns. This happens automatically during data ingestion and compaction—no configuration needed.

Let's compare the physical layout of data before and after dynamic shredding.

Before (Current):

After (With Dynamic Shredding):

This layout is generated dynamically for each Parquet file we write out based on the actual data we see. We don't have to include all null columns and we can prioritize shredding the most frequently seen or largest attributes.

Now when the query above runs:

SELECT * FROM events WHERE attributes->>'my_llm_response' like '%important%';

It can efficiently scan just the _lf_attributes_my_llm_response column:

┌─────────────────────────────────────────┐
│  Benefit 1: Index pruning               │
│  Zone maps and full text search indexes │
│  let us skip entire files / row groups  │
└─────────────────────────────────────────┘
                    ↓
┌──────────────────────────────────────────┐
│  Benefit 2: Download Only Relevant Column│
│  (_lf_attributes_my_llm_response)        │
└──────────────────────────────────────────┘
                    ↓
┌─────────────────────────────────────────┐
│  Benefit 3: Better Compression          │
│  English text string data               │
│  via zstd + dictionary encoding         │
│  Maybe FastLanes in future?             │
└─────────────────────────────────────────┘
                    ↓
┌─────────────────────────────────────────┐
│  Benefit 4: Zero JSON Parsing           │
│  Data already extracted & typed         │
│  No deserialization, zero copy into mem │
└─────────────────────────────────────────┘

This query now benefits from the full efficiency of progressive narrowing / pruning and columnar storage. It is as fast as if my_llm_response were a first-class column in the schema.

#The Hard Path to Dynamic Shredding

We've been working on this feature for nearly 6 months now. It's required a lot of upstream changes in DataFusion to enable expression pushdown and predicate pushdown for JSON accessors. This is the blessing and the curse of working with an upstream open source project: we get to build on a solid foundation, but we also have to wait for upstream changes to land and get released. We've worked on this feature in a way that benefits not only us but anyone using DataFusion with semi-structured or nested data, including Parquet/Arrow struct columns and the newly introduced Json Variant type.

We've had to refactor core parts of the DataFusion execution engine to push down expressions into the Parquet reader so that it can use the physical file layout to optimize expression evaluation (in our case to decide if we need to read from the _lf_attributes remainder column or if there is a shredded column we can use instead). This work is done and has fully landed as of DataFusion 52.1.0 (Jan 2026).

The remaining work involves actually getting these expressions into the scan operators, e.g. for cases like:

SELECT attributes->>'http.response.status_code', count(*)
FROM records
GROUP BY attributes->>'http.response.status_code';

Pulling out the attributes->>'http.response.status_code' expression from the GROUP BY clause and pushing it down into the Parquet scan so that we can use the shredded column directly is a bit tricky, and involves complex expression rewriting and optimization passes, but we expect to land this work in DataFusion by Feb 2026. The upshot of this is that we've been able to contribute major improvements to DataFusion that benefit a broad community of users.

#Minimizing Breakage for Users

Our plan to land this while minimizing bugs and breakage for users is to first rewrite our query path (non-destructive, most of this has already been done) to use dynamic shredding but keep the static shredding in the write path. Along with this change we'll eliminate and isolate all notion of hardcoded shredded columns from the codebase into a self-contained section of the write path. Essentially our initial heuristic for dynamic shredding will mimic the current static shredding configuration. This lets us test out the vast majority of these changes without making any irreversible changes (like writing out a new physical layout).

Once we have confidence that the dynamic shredding query path is solid, we can flip the switch in the write path to enable dynamic shredding using the discovery algorithm.

#How do we decide what to shred?

Analysis of our production data has shown that 99% of our existing Parquet files have fewer than 128 unique top-level keys in the attributes JSON column. The P50 is closer to 32 unique keys.

We've thus chosen a default of extracting the top 128 largest (by total size across all rows) keys from each JSON column during ingestion/compaction.

This automatic discovery approach balances the tradeoff between:

• Extracting enough attributes to cover common query patterns and maximize performance benefits
• Avoiding excessive column proliferation and overhead from shredding too many attributes

For example, if you accidentally start sending logs like logfire.span('foo', **{uuid.uuid4(): 'some value' for _ in range(1000)}), we don't want to shred all 1000 dynamically generated keys. This algorithm is robust to even such pathological cases because it will shred the non-unique keys that dominate the data while putting the rest of the "accidental" dynamically generated keys into the reduced JSON column. Queries against these rare keys will still work, just with the usual JSON parsing overhead. Queries against the rest of the data will remain fast.

Having a hard cap on the number of shredded columns lets us have a controlled overhead in terms of Parquet column count which avoids bloating the schema and incurring excessive overheads in parsing of Parquet metadata. Our empirical measurements shows that ~ 1000 columns is not a problem for Parquet / DataFusion, but more than that and you start to see a noticeable performance overhead.

#How are data types determined?

As we discover which attributes to shred, we also need to determine the data type for each shredded column. We do this by keeping track of the current planned output data type for each candidate attribute as we scan through the data and widening the type as needed. The basic rule is Int -> Float -> JSON, Bool -> JSON and String -> JSON. There is an important distinction between String and JSON. Consider the values [{"user_id": "123"}, {"user_id": 456}]. In this case we would store the data as JSON which practically speaking is a Utf8View column containing JSON strings: ["123", 456]. This means JSON parsing is still needed to read from this column but we still benefit from columnar storage, compression, and to some extent pruning. If on the other hand we have ["123", "456"] we can store this as a Utf8View column directly without JSON parsing: [123, 456].

#Future work

#Conclusion

In the next few months you'll probably see your Logfire queries get orders of magnitude faster. You don't need to do anything on your end, just continue to use Logfire as you currently do!

#References and related work

• DataFusion Blog: Parquet Pushdown
• ClickHouse JSON Type Reference
• Parquet Variant type