PyAI Berlin meetup with JetBrains & Pydantic, featuring GuruSup
An evening of talks at the JetBrains office in Berlin, hosted by Jodie Burchell and Laís Carvalho: Jodie on what LLM eval scores actually prove, Laís on what Code Mode does to a Pydantic AI agent, and Fran Pastor Agustí on turning company knowledge into agent skills with Pydantic AI and Monty. Food and drinks after.
Talks on LLM evals, Code Mode and agent skills at the JetBrains office in Berlin, with JetBrains, Pydantic and GuruSup. Food and drinks after.
Overview
- In-person event
- Berlin, Germany
- Free
Speakers
-
Jodie Burchell Team Lead of Python Advocacy · JetBrains Leads Python advocacy at JetBrains. Clinical psychology PhD and biostatistics postdoc turned data scientist, with over 10 years building recommendation systems, search and analysis platforms in Australia and Germany. -
Laís Carvalho Developer Relations · Pydantic Growth marketing & Developer Relations at Pydantic. Passionate about Python, open source, and building great developer experiences.
-
Fran Pastor Agustí CTO · GuruSup Psychologist turned software engineer with 13 years in full-stack development. At GuruSup he has spent three years building AI agents that use company knowledge to get work done.
About this event
Jodie Burchell (JetBrains) and Laís Carvalho (Pydantic) are hosting an evening PyAI Meetup at the JetBrains office in Berlin, with three talks from people building and measuring these systems, then food, drinks and time to argue about them.
The talks
Jodie Burchell, Team Lead of Python Advocacy at JetBrains: "Reliable, rigorous, wrong: A psychometric view of LLM evals". Held-out test sets, calibrated LLM judges and significance tests tell you a score really changed. They do not tell you the system got better. Jodie treats LLM evaluation as a measurement problem, one psychology has worked on for over a century, and applies the psychometric idea of validity: not only "is this score reliable?" but "what evidence supports the conclusion I want to draw from it?" She starts with how benchmarks fail basic validity checks, moves to the metrics teams build for their own applications, such as "helpfulness" or "task success", and closes with four validity questions to apply to your next eval.
Laís Carvalho, Developer Relations at Pydantic: "What one line of Code Mode does to your agent". If you've tried writing your own agents, you've probably tried a weather agent. My Pydantic AI weather agent has two tools, get_lat_lng and get_weather, and it does fine when I ask about two locations. So I gave it a harder question: which EU capital will be warmest this weekend? That's 27 cities and 54 tool calls, with every result going back through the model. It got there, slowly. In one of my test runs it decided 8 capitals were probably enough.
In this talk we'll add Code Mode to that agent. The model will write a short Python script that loops over all 27 cities. The script runs in Agent Sandboxes, the hosted Monty sandboxes in Pydantic Logfire, where it can only call the functions I hand it. Let's compare the runs in Logfire: tokens, cost, time, and latency using a sandbox.
Fran Pastor Agustí, CTO at GuruSup: "From Company Knowledge to Living Skills: GuruSup's Journey with Pydantic AI and Monty". Fran walks through an agent that investigates a customer's order sync incident across Gmail, MongoDB and Logfire, then records it in Notion. Both agents are built with Pydantic AI and return typed Pydantic results. The talk follows how the skill evolves. It starts as a Markdown procedure that guides the tool calls one by one. Then Code Mode lets the model write Python that chains the calls, running in the Pydantic Monty sandbox. A second agent, the improver, writes one program for the whole investigation, which is saved in the skill so later runs just execute it. When it fails, the agent falls back to the Markdown procedure and still finishes the task. The improver then repairs the program from the Logfire trace, and a person approves the fix as a new revision of the skill. He closes with how GuruSup does this behind the scenes, turning company knowledge into executable skills without users having to build and refine the workflow themselves.
Schedule
- 6:00pm Doors open
- 6:30pm Talks and Q&A
- 7:30pm Food, drinks and networking
- 9:00pm Close
Registration is on Luma. Spots are limited to 100, with a waitlist once they fill.
What you'll learn
- Four validity questions to ask before trusting your next eval score
- What one line of Code Mode changes in tokens, cost and time, measured in Logfire
- How an agent skill grows from a Markdown procedure into a saved program that is repaired from its Logfire trace
- How GuruSup turns company knowledge into executable skills with Pydantic AI and Monty
Save your spot
Registration is handled on luma.com, where the host confirms your place and shares the address.
Berlin, Germany
Register for the eventExplore Logfire
Explore our open source packages