The problem you will solve
A team wants to choose a model for extracting support-ticket facts. Build a reproducible comparison that can reveal a slower but more accurate configuration or a fast configuration that invents fields. Do not equate tokens per second with useful output.
Before you start
HTTP model calls, validation and evaluation-set design.
Suggested tech stack
Data plan
Write 40 fictional support tickets and label category, stated product and explicit urgency evidence. Split development and held-out cases before prompt tuning; include ambiguous tickets that should produce null fields.
Hardware and cost
Compare two configurations of a model that fits your hardware, or two permitted APIs. Run sequentially on limited RAM, cap tokens and trials, and note hardware and any charges. Large-model training is out of scope.
Define the input and output
Use this illustrative contract as a starting point, then specify required fields, validation and error responses in your own implementation. It is not a complete API schema.
Input: {"ticket":"The demo export fails for Atlas."}
Output: {"product":"Atlas","category":"export_failure","urgency":null}
Run record: config_id, prompt_hash, duration_ms, schema_valid, field_scoresYour five build milestones
Finish and test one milestone before extending the system. Keep an experiment log with the decision, the alternative you rejected and the evidence behind the choice.
Freeze the task
Define field labels, ambiguity rules and an expected-output fixture. Keep product names that are absent from the ticket null rather than guessing from prior examples.
Build one adapter
Normalize request, timeout and response logging across configurations. Store errors and raw outputs; do not drop failures from the comparison denominator.
Separate warm-up
Measure loading and the first request separately from repeated requests. Run configurations in alternating order when possible to reduce temperature or background-load bias.
Score facts and format
Validate JSON, then compare factual fields against the held-out labels. Count unsupported values and produce a confusion table for category errors.
Publish a decision card
Show per-configuration accuracy, failure counts and latency distribution with sample size. Explain the chosen trade-off and include a one-command rerun with credentials supplied through environment variables.
How to evaluate your project
Set your acceptance criteria before looking at the final test results. The checks below are proposed experiments, not measured outcomes or a guarantee that a particular model will pass.
- Report schema validity, exact-match field accuracy and unsupported-value rate independently.
- Record median and p95 latency with run counts; label tiny-sample estimates as unstable.
- Rerun identical inputs and show whether output changes. Log timeouts as failures rather than fast successes.
Save your test cases, run command, dependency versions and results together. Compare on identical inputs, retain failures in the denominator and report sample sizes. If a change improves one metric but harms another, explain which trade-off fits your use case.
Failure modes worth demonstrating
- Tuning on the held-out cases or comparing prompts that ask for different outputs.
- Publishing API secrets in logs, or counting a fast malformed answer as useful throughput.
Optional extension
Add a quality gate that fails a candidate configuration when unsupported values increase, even if average latency improves.
Turn the build into portfolio evidence
Your repository should let another person understand the problem and reproduce the core evaluation without needing your private credentials.
- A README with problem, intended user, architecture, setup commands and a small input/output example.
- A permitted sample dataset, its origin or generator, a frozen test split and a single evaluation command.
- A results table comparing your baseline with your final approach, including failures and sample counts.
- A short demo or screenshots, a limitations section and one decision you would change with more time.
- Environment-variable names without secret values, a dependency list and attribution/license notices for any reused code or datasets.
Resume framing
Describe this as a personal or educational project. State what you implemented, how you tested it and what your measured results were. Replace placeholders with your own evidence; do not claim employer impact, production traffic or accuracy you did not measure.
Built llm structured-output comparison lab using Python, JSON Schema, SQLite; evaluated against [baseline] on [N held-out cases], documented [measured result] and [important limitation].Be ready to explain your decisions
- What makes your comparison fair?
- When would you choose the slower configuration?
Answer with a concrete failure from your implementation, the options you considered, your measured evidence and the limitation that remains.
Google documentation and GitHub references
The project scope, milestones and evaluation plans above are original AIJobFever educational material. These official Google resources offer background or implementation examples; they are not finished implementations of this blueprint. No Google endorsement is implied. Review repository licenses, supported platforms and current costs before reusing code or provisioning services.
Our editorial policy