The problem you will solve
An analyst wants a first draft of a weekly product report from supplied notes. Build a tool that attaches evidence to factual statements and separates observations from suggested actions. It should say there is insufficient evidence when a requested section cannot be supported.
Before you start
Basic prompting, JSON validation and simple Python interfaces.
Suggested tech stack
Data plan
Author eight fictional weekly evidence packs with stable document IDs. Include contradictory notes, outdated figures and a week with no usage data. Label which statements each document supports.
Hardware and cost
Use a small evidence pack and capped output length. Hosted models may charge per request; local models need adequate RAM. Keep a fixture-based demo usable without an API key.
Define the input and output
Use this illustrative contract as a starting point, then specify required fields, validation and error responses in your own implementation. It is not a complete API schema.
Input: {"week":"W1","documents":[{"id":"note-1","text":"Demo signups were 18."}]}
Output: {"observations":[{"text":"18 demo signups were recorded.","sources":["note-1"]}],"unknowns":["Conversion rate unavailable"]}Your five build milestones
Finish and test one milestone before extending the system. Keep an experiment log with the decision, the alternative you rejected and the evidence behind the choice.
Set evidence rules
Define an output schema for observations, recommendations and unknowns. Require every observation to reference supplied IDs; keep advice explicitly distinct from factual findings.
Create a deterministic draft
Render the evidence as a simple template before adding generation. Use this baseline to judge whether fluent wording adds anything useful.
Generate and validate
Keep source text separate from instructions. Check that every cited ID exists and reject uncited factual sections. Record prompt and model versions for each run.
Review claim support
Manually judge whether each cited passage actually supports the claim. A matching ID is not enough. Show conflicting figures with dates instead of choosing one silently.
Build a review screen
Display draft claims beside their evidence. Export only after a human review step; publish examples of abstention and a documented unsupported-claim failure.
How to evaluate your project
Set your acceptance criteria before looking at the final test results. The checks below are proposed experiments, not measured outcomes or a guarantee that a particular model will pass.
- Measure supported factual claims divided by all factual claims using a written reviewer rubric.
- Report citation-ID validity separately from human-judged support and coverage.
- Compare two prompts on the same packs for support, edit effort, latency and recorded token use.
Save your test cases, run command, dependency versions and results together. Compare on identical inputs, retain failures in the denominator and report sample sizes. If a change improves one metric but harms another, explain which trade-off fits your use case.
Failure modes worth demonstrating
- Presenting plausible recommendations as measured facts.
- Assuming a citation guarantees entailment or feeding private business notes to a public model.
Optional extension
Add a claim-level change view so a reviewer sees exactly what changed between two weekly drafts.
Turn the build into portfolio evidence
Your repository should let another person understand the problem and reproduce the core evaluation without needing your private credentials.
- A README with problem, intended user, architecture, setup commands and a small input/output example.
- A permitted sample dataset, its origin or generator, a frozen test split and a single evaluation command.
- A results table comparing your baseline with your final approach, including failures and sample counts.
- A short demo or screenshots, a limitations section and one decision you would change with more time.
- Environment-variable names without secret values, a dependency list and attribution/license notices for any reused code or datasets.
Resume framing
Describe this as a personal or educational project. State what you implemented, how you tested it and what your measured results were. Replace placeholders with your own evidence; do not claim employer impact, production traffic or accuracy you did not measure.
Built evidence-backed report drafting using Python, JSON Schema, SQLite; evaluated against [baseline] on [N held-out cases], documented [measured result] and [important limitation].Be ready to explain your decisions
- How do you evaluate a draft without a single perfect answer?
- What happens when two sources disagree?
Answer with a concrete failure from your implementation, the options you considered, your measured evidence and the limitation that remains.
Google documentation and GitHub references
The project scope, milestones and evaluation plans above are original AIJobFever educational material. These official Google resources offer background or implementation examples; they are not finished implementations of this blueprint. No Google endorsement is implied. Review repository licenses, supported platforms and current costs before reusing code or provisioning services.
Our editorial policy