LLM-as-a-Judge: A Practical Guide with Pydantic Evals

#One-Size-Fits-All Evaluators

A one-size-fits-all evaluator uses a single rubric applied uniformly to every test case:

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import LLMJudge

dataset = Dataset(
    cases=[
        Case(name='billing_question', inputs='Why was I charged twice?'),
        Case(name='feature_request', inputs='Can you add dark mode?'),
        Case(name='bug_report', inputs='The app crashes when I upload photos'),
    ],
    # This evaluator runs on ALL cases with the same rubric
    evaluators=[
        LLMJudge(
            rubric='Response is professional, empathetic, and does not blame the user',
            include_input=True,
        ),
    ],
)

The rubric must be general enough to apply everywhere. This works well for quality dimensions that genuinely are universal: professionalism, safety, tone. These are especially useful for evaluating orthogonal concerns, things that should always be true regardless of what the LLM was actually asked to do.

#Case-Specific Evaluators

A case-specific evaluator has a rubric tailored to an individual test case:

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import LLMJudge

dataset = Dataset(
    cases=[
        Case(
            name='vegetarian_recipe',
            inputs='I need a vegetarian dinner recipe',
            evaluators=[
                LLMJudge(
                    rubric='''
                    Recipe must NOT contain meat, meat-based broths,
                    gelatin, or other animal-derived ingredients.
                    PASS only if the recipe is fully vegetarian.
                    ''',
                    include_input=True,
                ),
            ],
        ),
        Case(
            name='quick_weeknight_meal',
            inputs='I need something I can make in under 30 minutes',
            evaluators=[
                LLMJudge(
                    rubric='''
                    Recipe must have total prep + cook time under 30 minutes,
                    use commonly available ingredients, and not require
                    specialized equipment.
                    FAIL if the recipe would realistically take longer.
                    ''',
                    include_input=True,
                ),
            ],
        ),
        Case(
            name='allergy_safe',
            inputs='Nut-free dessert for a school event',
            evaluators=[
                LLMJudge(
                    rubric='''
                    Recipe must NOT contain tree nuts, peanuts, or
                    peanut-derived ingredients. This is for a child
                    with allergies. FAIL if any nut risk exists.
                    ''',
                    include_input=True,
                ),
            ],
        ),
    ],
    # Universal evaluator still runs on all cases
    evaluators=[
        LLMJudge(
            rubric='Recipe instructions are clear and easy to follow',
            include_input=True,
        ),
    ],
)

Notice how each case has requirements that would be impossible to squeeze into a single universal rubric. The vegetarian case couldn't care less about cooking time. The quick meal case has no opinions about allergens. Trying to write one rubric that captures all of this would produce something vague and difficult to maintain. You'd end up spending real effort evaluating whether your evaluator is even working correctly. With case-specific rubrics, the rubric itself is simple enough that you can trust it on inspection.

#Why This Matters

Here's the realization that ties it together:

If an LLM were good enough to assess quality reliably across all cases without context, it would likely be good enough to generate good responses in the first place.

This is especially true when evaluating agents that can reflect on their own output, or when using thinking models for generation. In those cases, a generic LLM rubric would only catch failures the model could have avoided by thinking one step longer. Case-specific evaluators sidestep this by providing context the judge needs that the generator never had:

• One-size-fits-all: "Is this a good essay?" (vague, hard to judge)
• Case-specific: "Does this essay argue that renewable energy is cost-effective, using at least three economic studies?" (clear, verifiable)

That said, one-size-fits-all evaluators have a compelling use case beyond style and safety checks: validating cheaper or faster models. If you're using a smaller model in production to save cost or reduce latency, a more capable judge model can confirm the cheaper one is producing adequate results. For many tasks, a flagship model is overkill, but it's good to verify that. Running a capable judge over a sample of your cheaper model's outputs is a practical way to gain that confidence.

#When to Use Each

#Groundedness and Hallucination Detection

LLM judges are particularly effective at detecting hallucinations because the task is well-defined: "Is claim X supported by source Y?"

from pydantic import BaseModel
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import LLMJudge


class RAGInput(BaseModel):
    question: str
    retrieved_context: str


dataset = Dataset(
    cases=[
        Case(
            name='policy_question',
            inputs=RAGInput(
                question='What is the return policy?',
                retrieved_context='''
                Items can be returned within 30 days of purchase.
                Items must be unused and in original packaging.
                Refunds are processed within 5-7 business days.
                Sale items are final sale.
                ''',
            ),
            evaluators=[
                LLMJudge(
                    rubric='''
                    The response must ONLY contain information from the
                    retrieved context. FAIL if the response includes ANY
                    information not in the context — even if it seems plausible.
                    ''',
                    include_input=True,
                ),
            ],
        ),
    ],
)

The include_input=True is crucial here. It gives the judge the source material to check against. Without it, you're asking someone to spot lies without telling them what the truth is.

#Style and Tone

Style checks cross-cut other quality dimensions and make excellent one-size-fits-all evaluators:

from pydantic_evals.evaluators import LLMJudge

style_evaluators = [
    LLMJudge(
        rubric='Response uses second person ("you", "your") instead of third person',
        assertion={'evaluation_name': 'uses_second_person'},
    ),
    LLMJudge(
        rubric='''Response avoids corporate jargon: "leverage", "synergy",
        "paradigm", "circle back", "move the needle", etc.''',
        assertion={'evaluation_name': 'no_jargon'},
    ),
]

These are clear, binary criteria that apply universally and are hard to check programmatically (regex would be brittle).

#Hard-to-Generalize Quality Rules

Some criteria are easy to articulate for a specific case but hard to write as universal rules:

• "This coding question needs a solution that handles the edge cases the user mentioned"
• "This customer is frustrated, so the response needs to be especially empathetic"
• "This is a follow-up question, so the response shouldn't repeat context already established"

Case-specific LLM judges handle these naturally.

#When Deterministic Checks Will Do

This should go without saying, but: if you can check it with code, check it with code.

• Type validation: Use Pydantic models
• Format validation: JSON schema, regex patterns
• Length constraints: Character or word counts
• Exact value matching: String equality, numeric comparisons

Deterministic checks are faster (milliseconds vs. seconds), cheaper (free vs. API calls), perfectly consistent, and much easier to debug.

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import LLMJudge, IsInstance

dataset = Dataset(
    cases=[
        Case(name='refund_request', inputs='I want a refund for my order'),
    ],
    evaluators=[
        # Fast, deterministic check first
        IsInstance(type_name='str'),
        # Semantic check that needs LLM judgment
        LLMJudge(
            rubric='Response acknowledges the refund request and provides next steps',
            include_input=True,
        ),
    ],
)

Structure your evaluators from cheapest to most expensive. If the response isn't even the right type, there's no point asking a language model whether it's helpful.

#The Nuance: Brittle vs. Robust Matching

Checking for specific words feels deterministic but is often brittle. A naive 'refund' in response check fails because of synonyms ("reimbursement", "money back"), negations ("We cannot offer a refund" contains "refund" but means the opposite), and context differences. For semantic matching, LLM judges are more robust.

But sometimes exact matching is exactly right: verifying a calculated value appears verbatim, checking for required legal disclaimers, or ensuring a specific product SKU is mentioned.

#Binary vs. Numeric

Use binary (pass/fail) when the world really is black and white: policy compliance, safety violations, the presence or absence of PII. These are not matters of degree.

Use numeric scores when quality exists on a spectrum, you want to track improvement over time, or you need to rank responses.

from pydantic_evals.evaluators import LLMJudge

# Binary assertion
assertion_judge = LLMJudge(
    rubric='Response follows company guidelines',
    assertion={'evaluation_name': 'compliant', 'include_reason': True},
    score=False,
)

# Numeric score
score_judge = LLMJudge(
    rubric='Rate the helpfulness of the response from 0 to 1',
    score={'evaluation_name': 'helpfulness', 'include_reason': True},
    assertion=False,
)

#Context Levels

Three levels of context you can give the judge:

• Output only (default): For style checks and format validation
• Input + Output (include_input=True): For relevance, completeness, appropriateness
• Input + Output + Expected (include_input=True, include_expected_output=True): For correctness and semantic equivalence

Give the judge the minimum context it needs. More context can actually hurt by distracting from the criteria that matter.

#Always Request Reasoning

from pydantic_evals.evaluators import LLMJudge

judge = LLMJudge(
    rubric='Response is accurate',
    assertion={'evaluation_name': 'accurate', 'include_reason': True},
)

When a case fails, the reason tells you why. Without it, you're staring at a red light with no idea what tripped it. The reasoning lets you debug, verify the judge actually applied your rubric correctly (they don't always), and iterate on your rubrics based on how the judge interprets them.