Compare AI Models on Your Own Evaluation Set

A model that performs well on a public benchmark may still struggle with your support tickets, extraction rules, or coding conventions. The most useful comparison starts with examples from your own workflow. This walkthrough shows you how to build a small, repeatable evaluation and use it to make a practical decision.

#1 · Define the decision before choosing models

Write down what you need the model to do and what would make its output acceptable. For invoice extraction, that might mean returning valid structured data with the correct supplier, currency, and total. For support replies, it might mean answering accurately without inventing a refund policy.

Separate essential requirements from preferences. Correctness might be mandatory, while a warmer tone is merely desirable. This prevents an impressive writing style from hiding a serious factual mistake.

#2 · Collect a small, representative evaluation set

Start with a few dozen examples you can inspect yourself. Include common requests, difficult cases, incomplete inputs, and situations where the correct behavior is to ask a question or decline to guess. Remove personal information, credentials, and confidential material before sending data to any external service.

For each example, save an identifier, the input, an expected answer or checklist, and a category. Keep a separate holdout set that you do not use while adjusting prompts. Otherwise, you risk optimizing for familiar examples rather than improving general performance.

#3 · Write the scoring rubric first

Make the rubric specific enough that another person could apply it consistently. A simple starting point is:

• Correctness: Does the answer satisfy the reference facts or required checks?
• Instruction following: Does it respect the requested format and constraints?
• Grounding: Does it avoid unsupported claims and handle missing information appropriately?
• Operational fit: Does it meet your measured cost and latency limits?

Use automated checks for properties such as valid JSON, required fields, or passing tests. Use human review for qualities that those checks cannot establish. If another model helps grade answers, validate its judgments against human-reviewed examples rather than treating its scores as ground truth.

#4 · Choose candidates and standardize the setup

For an initial comparison, you could test Claude Haiku 4.5 and Gemini 2.5 Flash. Treat them as candidates, not predetermined winners. Check availability, supported features, pricing, and exact API identifiers before running the evaluation.

Relay365 is a pay-as-you-go relay desk for multiple model families, making it a practical option when you want to compare candidates through a common access point. Its API base is https://api.api365relay.com; OpenAI SDK clients add /v1. A shared interface reduces integration work, but does not make every model's parameters or behavior identical.

#5 · Run the same tasks under comparable conditions

Use the same task instructions, input data, and output requirements for each candidate. Keep output limits and sampling settings comparable where supported, and record any differences. Fix tool access and retrieval inputs so you are not accidentally comparing different sources of information.

Save the exact request, response, model identifier, settings, timestamp, elapsed time, reported token usage, and errors. Define whether latency includes retries. Repeat important or unstable cases because even low-randomness settings do not guarantee identical answers.

#6 · Review outputs without model labels

Shuffle the answers and hide their model names during review. Score them using your rubric, then compare results on the same examples. This paired approach makes it easier to spot cases where one candidate succeeds and another fails.

Inspect failures by category, not just overall score. A strong average can conceal poor performance on missing information or strict formatting. With a small dataset, treat narrow differences as tentative rather than decisive.

#7 · Compare quality, cost, and latency together

Calculate costs using recorded usage and applicable prices, including retries and any additional billed features. Compare typical latency alongside slower cases. A cheaper request may not produce a cheaper completed task if it requires corrections. Set minimum quality requirements first, then choose among the candidates that meet them.

#8 · Preserve the evaluation and rerun it

Version your examples, rubric, prompts, and results. Add newly discovered failures, while protecting an untouched holdout set. Rerun the comparison after meaningful model, prompt, or workflow changes.

If a shared relay fits your testing workflow, create a Relay365 account and start with a small, budget-capped evaluation. Choose the model that best meets your measured requirements, not the one with the most persuasive demonstration.