Pixeltable
Pixeltable gives an agent read-only tools over existing Pixeltable tables
and views: list_tables, describe_table, query_table (equality filters), and similarity_search
over an embedding index. PixeltableMemoryStore keeps Memory in a Pixeltable table, so
the agent’s notes sit in the same catalog as the data it searches. Each works without the other.
Catalog source and memory store source
While Pydantic AI Harness is on 0.x releases, the API may change between minor releases; when it does, deprecation warnings and release-note migration guidance tell you (or your agent) exactly how to upgrade. See the version policy.
Install the pixeltable extra (Python 3.11 to 3.14; Pixeltable 0.7.21 or later). The example uses an OpenAI model and OpenAI
embeddings, so it also pulls in the openai provider, plus spec for the YAML agent spec below:
pip install "pydantic-ai-harness[pixeltable]" "pydantic-ai-slim[openai,spec]"
uv add "pydantic-ai-harness[pixeltable]" "pydantic-ai-slim[openai,spec]"
A table with an embedding index, created once (in an application, declare it on a Pixeltable
TableModel and apply it with pxt schema update):
import pixeltable as pxt
from pixeltable.functions.openai import embeddings
pxt.create_dir('hr', if_exists='ignore')
handbook = pxt.create_table('hr.handbook', {'topic': pxt.String, 'text': pxt.String}, if_exists='ignore')
handbook.insert([{'topic': 'expenses', 'text': 'Meals during business travel are reimbursed up to 60 EUR per day.'}])
handbook.add_embedding_index('text', embedding=embeddings.using(model='text-embedding-3-small'), if_exists='ignore')
An agent that searches it and keeps notes across runs:
from pydantic_ai import Agent
from pydantic_ai_harness import Memory, Pixeltable
from pydantic_ai_harness.memory import PixeltableMemoryStore
agent = Agent(
'openai:gpt-6.1-sol',
capabilities=[
Pixeltable(['hr.handbook']),
Memory(PixeltableMemoryStore(table_name='hr.memory')),
],
)
result = agent.run_sync('Can I expense a 75 EUR dinner? Remember that I travel monthly.')
print(result.output)
#> ...
Download the video example
and run it once with a short expense-policy video containing speech and OPENAI_API_KEY set:
python pixeltable_video.py training.mp4
uv run python pixeltable_video.py training.mp4
On insertion, Pixeltable calls OpenAI to transcribe 30-second audio segments, caption one frame every 10 seconds, and index the stored text. Searches embed the query and retrieve that text with timestamps; media columns are omitted. The agent also saves a preference for visual examples and recalls it in a second run without the first run’s chat history.
| Tool | What it does |
|---|---|
list_tables | The allowed table and view paths. |
describe_table | Kind, comment, columns (type, is_computed, is_stored), and indexes. |
query_table | Rows matching equality filters ({"status": "open"}); timestamp, date, and UUID values are ISO strings. |
similarity_search | Nearest rows by column.similarity(string=query), with a similarity score in the result. Each call embeds the query with the index’s embedding function, which may be a paid provider call. |
tablesrequires table paths or directory prefixes;['*']allows the whole catalog, including memory tables. Allowed views expose their base table’s columns. Version handles ('dir.tbl:3') are refused because they expose deleted rows and dropped columns.- Default projections skip media, array, binary, unstored computed columns, and unstored iterator
outputs, including indexed media in similarity search. Explicit projections reject the same
columns: media because it exposes local file paths, and unstored computed or iterator columns
because they rerun their functions (possibly model calls) on read. Filters reject the same
columns, including
Nonefilters on media, which would open every file before filtering. query_tableandsimilarity_searchreturn{"table", "rows", "truncated"}, bounded bymax_rows(default 20) andmax_chars(default 8000). Oversized strings end in...; other oversized values becomenull, andtruncatedreports applied bounds. A cap below the minimal result envelope still returns that envelope.list_tablesanddescribe_tableare sized by the allowlist and schema.- Invalid requests, missing tables, and transient provider errors become
ModelRetry; infrastructure and authorization failures propagate to the application. - The default instructions tell the model to describe unfamiliar tables first and to treat table
contents as untrusted data. Pass
guidance='...'to replace them, orguidance=''to add none.
PixeltableMemoryStore(table_name='harness.memory') implements MemoryStore and
SearchableMemoryStore. The table is created on first use.
- Writes use compare-and-set versions and operation receipts, so a retried tool call replays its
mutation instead of applying it twice. Paths are limited to 255 characters and operation ids to
248; the
__op__path root is reserved. - Pixeltable has no multi-statement transaction, so a receipt is recorded as pending before its mutation and completed after it, and receipts are never deleted. Every attempt of one operation writes the same version, so its receipt matches whichever attempt lands. Deleting a file keeps an empty marker row that names the deleting operation. The next mutation of a file completes the pending receipts whose write or delete is visible there; any other waits for its operation’s next attempt.
- Compare-and-set and replay hold with several writers, in one process or several, and with a crash between any two table operations: an operation is applied at most once, and its retry replays it.
- It does not implement
MetadataMemoryStore, soMemory(write_metadata=...)raisesUserError. search_memoryuses the same lexical scoring and prefix isolation as the other stores.store.tablesupports queries, joins, and computed columns; filter tokind == 'file'for memory rows. Route writes and deletes through the store to preserve versions and receipts.- An uninterrupted write with a new operation id creates three table versions (pending receipt, apply write, complete receipt). Receipts are retained for replay; deleting them would not remove their history, so the table grows with both operations and row versions.
- Deleted or overwritten memories stay readable through earlier table versions (for example
pxt.get_table('hr.memory:3')). Pixeltable has no API to prune history, so the only way to purge a memory is to drop the table. Do not store data in it that must be erasable. - Each insert prints a status line such as
Inserted 1 row with 0 errors; setPIXELTABLE_VERBOSITY=0to silence it in CLIs and TUIs.
Two Pixeltable capabilities on one agent share id='pixeltable'. The same configuration stated
twice is one; two that differ raise UserError, because tables is an access boundary. Give each
its own id and wrap them in PrefixTools to keep both. A capability passed to a single run
replaces the agent’s instead of merging with it, so it can widen what that run reaches.
The package adds no spans; core records tool calls and Memory emits memory.* spans.
Pixeltable works with Pydantic AI’s agent spec. The allowlist is
its one positional argument, so the short form passes it alone:
# agent.yaml
model: openai:gpt-6.1-sol
capabilities:
- Pixeltable: ['hr.handbook']
from pydantic_ai import Agent
from pydantic_ai_harness import Pixeltable
agent = Agent.from_file('agent.yaml', custom_capability_types=[Pixeltable])
Pass custom_capability_types so the spec loader knows how to instantiate it.
PixeltableMemoryStore is Python-only; the Memory spec backends are memory, file, and sqlite.
Bases: AbstractCapability[AgentDepsT]
Read-only tools over existing Pixeltable tables.
Pair this with Harness Memory (any store) when the agent also keeps a notebook.
from pydantic_ai import Agent
from pydantic_ai_harness.pixeltable import Pixeltable
agent = Agent(
'openai:gpt-5.6-sol',
capabilities=[Pixeltable(tables=['my_app.doc_chunks'])],
)
Allowlist of table paths or directory prefixes. ['*'] is the whole catalog.
The only positional argument, so the spec short form {"Pixeltable": ["my_app.doc_chunks"]} works.
Hard cap on rows returned by query_table and similarity_search.
Type: int Default: DEFAULT_MAX_ROWS
Cap on serialized JSON characters for those two tools. The minimal
{"table", "rows", "truncated"} envelope is always returned, even when a
cap below its size is configured.
Type: int Default: DEFAULT_MAX_CHARS
System-prompt text. None uses the default; '' adds none.
Type: str | None Default: None
Capability id. Distinct from Harness Memory (id='memory').
@classmethod
def combine(
cls,
capabilities: Sequence[AbstractCapability[AgentDepsT]],
) -> AbstractCapability[AgentDepsT]
tables is an access boundary, so two instances that disagree raise instead of merging.
AbstractCapability[AgentDepsT]
Bases: FunctionToolset[AgentDepsT]
List, describe, query, and similarity-search Pixeltable tables.
def list_tables() -> TableNames
List the allowed Pixeltable table and view paths.
TableNames
def describe_table(table: str) -> TableDescription
Describe a table’s columns and indexes.
TableDescription — Kind, comment, columns, and indexes.
table : str
Pixeltable table path (for example my_app.doc_chunks).
def query_table(
table: str,
columns: list[str] | None = None,
where: dict[str, WhereValue] | None = None,
limit: int = 10,
) -> RowsResult
Select rows from a table. where is equality only ({"status": "open"}).
RowsResult — Matching rows, with truncated set when the row or character cap applied.
table : str
Pixeltable table path.
Columns to return. Omit to skip media, array, and binary columns. Explicit media, array, binary, and unstored computed or iterator columns are rejected.
Equality filters mapping column name to value; None matches null rows.
Media, array, and binary columns reject filters.
limit : int Default: 10
Maximum rows to return, capped by the capability.
def similarity_search(
table: str,
query: str,
column: str,
columns: list[str] | None = None,
limit: int = 5,
idx: str | None = None,
) -> RowsResult
Nearest-neighbor search on a column that has an embedding index.
RowsResult — Rows ordered by similarity, each with a score field. If the table has a column by
RowsResult — that name, the field takes similarity_ prefixes until it is free.
table : str
Pixeltable table path.
query : str
Text to embed and search for.
column : str
Column with an embedding index.
Extra columns to return with the match and score. Omit to return the default columns; indexed media is never returned. Explicit media, array, binary, and unstored computed or iterator columns are rejected.
limit : int Default: 5
Maximum rows to return, capped by the capability.
Embedding index name. Required when the column has more than one index.
Pydantic AI Harness MemoryStore persisted in a Pixeltable table.
Implements MemoryStore and SearchableMemoryStore. File mutations use
compare-and-set on the file row. Operation receipts live in the same table
under __op__/, so paths whose first segment is __op__ are reserved.
Pixeltable has no multi-statement transaction, so a receipt is written as
pending before its mutation and completed after it, and is never deleted.
Every attempt of one operation writes the same version, derived from the
operation id, so one receipt matches whichever attempt lands. A deleted
file keeps its row as a marker naming the deleting operation, so a delete
stays visible after the fact like a write’s version does. The next mutation
of a file completes the pending receipts whose mutation is visible there;
any other stays pending for its operation’s next attempt.
An uninterrupted write with a new operation id creates three table versions (pending receipt, apply write, complete receipt). Receipts are retained for replay; deleting them would not remove their history from Pixeltable.
table_name : str Default: 'harness.memory'
Pixeltable table path (e.g. 'harness.memory').
Underlying Pixeltable table for computed columns and queries.
Type: pxt.Table