Skip to main content
All events

PyAI Berlin meetup with JetBrains & Pydantic, featuring GuruSup

An evening of talks at the JetBrains office in Berlin, hosted by Jodie Burchell and Laís Carvalho: Jodie on what LLM eval scores actually prove, Laís on what Code Mode does to a Pydantic AI agent, and Fran Pastor Agustí on turning company knowledge into agent skills with Pydantic AI and Monty. Food and drinks after.

Talks on LLM evals, Code Mode and agent skills at the JetBrains office in Berlin, with JetBrains, Pydantic and GuruSup. Food and drinks after.

Overview

  • In-person event
  • Berlin, Germany
  • Free

Speakers

About this event

Jodie Burchell (JetBrains) and Laís Carvalho (Pydantic) are hosting an evening PyAI Meetup at the JetBrains office in Berlin, with three talks from people building and measuring these systems, then food, drinks and time to argue about them.

Jodie Burchell, Team Lead of Python Advocacy at JetBrains: "Reliable, rigorous, wrong: A psychometric view of LLM evals". Held-out test sets, calibrated LLM judges and significance tests tell you a score really changed. They do not tell you the system got better. Jodie treats LLM evaluation as a measurement problem, one psychology has worked on for over a century, and applies the psychometric idea of validity: not only "is this score reliable?" but "what evidence supports the conclusion I want to draw from it?" She starts with how benchmarks fail basic validity checks, moves to the metrics teams build for their own applications, such as "helpfulness" or "task success", and closes with four validity questions to apply to your next eval.

Laís Carvalho, Developer Relations at Pydantic: "What one line of Code Mode does to your agent". If you've tried writing your own agents, you've probably tried a weather agent. My Pydantic AI weather agent has two tools, get_lat_lng and get_weather, and it does fine when I ask about two locations. So I gave it a harder question: which EU capital will be warmest this weekend? That's 27 cities and 54 tool calls, with every result going back through the model. It got there, slowly. In one of my test runs it decided 8 capitals were probably enough.

In this talk we'll add Code Mode to that agent. The model will write a short Python script that loops over all 27 cities. The script runs in Agent Sandboxes, the hosted Monty sandboxes in Pydantic Logfire, where it can only call the functions I hand it. Let's compare the runs in Logfire: tokens, cost, time, and latency using a sandbox.

Fran Pastor Agustí, CTO at GuruSup: "From Company Knowledge to Living Skills: GuruSup's Journey with Pydantic AI and Monty". Fran walks through an agent that investigates a customer's order sync incident across Gmail, MongoDB and Logfire, then records it in Notion. Both agents are built with Pydantic AI and return typed Pydantic results. The talk follows how the skill evolves. It starts as a Markdown procedure that guides the tool calls one by one. Then Code Mode lets the model write Python that chains the calls, running in the Pydantic Monty sandbox. A second agent, the improver, writes one program for the whole investigation, which is saved in the skill so later runs just execute it. When it fails, the agent falls back to the Markdown procedure and still finishes the task. The improver then repairs the program from the Logfire trace, and a person approves the fix as a new revision of the skill. He closes with how GuruSup does this behind the scenes, turning company knowledge into executable skills without users having to build and refine the workflow themselves.

  • 6:00pm Doors open
  • 6:30pm Talks and Q&A
  • 7:30pm Food, drinks and networking
  • 9:00pm Close

Registration is on Luma. Spots are limited to 100, with a waitlist once they fill.

What you'll learn

  • Four validity questions to ask before trusting your next eval score
  • What one line of Code Mode changes in tokens, cost and time, measured in Logfire
  • How an agent skill grows from a Markdown procedure into a saved program that is repaired from its Logfire trace
  • How GuruSup turns company knowledge into executable skills with Pydantic AI and Monty

Save your spot

Registration is handled on luma.com, where the host confirms your place and shares the address.

Berlin, Germany

Register for the event

Get started with Logfire for free

Sign up today and see your first trace in under 2 minutes

01 Install

$ uv add logfire

Resolved 7 packages in 128ms
Installed 7 packages in 89ms
+ logfire

02 Instrument

import logfire

logfire.configure()
logfire.info('Hello {name}', name='world')

Free up to 10m records per month No credit card needed