The problem you will solve
A fictional company has public help notes and internal team notes. Build an assistant that answers only from documents the current user may access. Retrieval quality is important, but preventing cross-team leakage is a hard requirement, not a bonus accuracy metric.
Before you start
Retrieval basics, document chunking, HTTP authorization and model prompting.
Suggested tech stack
Data plan
Author 24 fictional policy notes for two teams, with document IDs, versions and access labels. Prepare 30 questions including unanswerable, conflicting-version and unauthorized-document cases.
Hardware and cost
Start with local keyword retrieval and a tiny corpus. Add embeddings and generation only after access tests pass. Hosted embedding/generation calls may incur cost; never use real internal documents in a public demo.
Define the input and output
Use this illustrative contract as a starting point, then specify required fields, validation and error responses in your own implementation. It is not a complete API schema.
Input: authenticated principal + question
Output: {"answer":"...","citations":["public-03#chunk-2"],"abstained":false}
Never accept a client-supplied team name as authorization.Your five build milestones
Finish and test one milestone before extending the system. Keep an experiment log with the decision, the alternative you rejected and the evidence behind the choice.
Model the boundary
Create server-controlled principals and document access labels. Write a permission matrix before adding a vector index; distinguish authentication from permission to read each source.
Build a retrieval baseline
Split notes by headings with stable chunk IDs and version metadata. Search only authorized content and compare keyword results with an embedding variant on the same questions.
Generate with provenance
Give the model only allowed chunks and require citations to supplied IDs. Add an insufficient-evidence response; treat retrieved instructions as data, not tool commands.
Attack the pipeline
Test direct requests for another team's note, malicious text inside a document and stale-version collisions. Check caches and logs as well as the answer body for leakage.
Explain retrieval failures
Save question, expected source, retrieved IDs and answer judgment. Demonstrate one missed retrieval and one unsupported answer, then document the change and retest.
How to evaluate your project
Set your acceptance criteria before looking at the final test results. The checks below are proposed experiments, not measured outcomes or a guarantee that a particular model will pass.
- Measure retrieval recall@5 against labeled supporting chunks before judging generation.
- Report citation support and correct abstention on unanswerable questions using a written rubric.
- Require zero unauthorized chunks in model inputs, outputs and shared caches on your permission tests; passing a small suite is not a security certification.
Save your test cases, run command, dependency versions and results together. Compare on identical inputs, retain failures in the denominator and report sample sizes. If a change improves one metric but harms another, explain which trade-off fits your use case.
Failure modes worth demonstrating
- Filtering access only after unrestricted documents have reached the model.
- Reusing a response cache across principals or assuming high similarity means factual support.
Optional extension
Add document revocation and prove that revoked sources disappear from retrieval and cached answers.
Turn the build into portfolio evidence
Your repository should let another person understand the problem and reproduce the core evaluation without needing your private credentials.
- A README with problem, intended user, architecture, setup commands and a small input/output example.
- A permitted sample dataset, its origin or generator, a frozen test split and a single evaluation command.
- A results table comparing your baseline with your final approach, including failures and sample counts.
- A short demo or screenshots, a limitations section and one decision you would change with more time.
- Environment-variable names without secret values, a dependency list and attribution/license notices for any reused code or datasets.
Resume framing
Describe this as a personal or educational project. State what you implemented, how you tested it and what your measured results were. Replace placeholders with your own evidence; do not claim employer impact, production traffic or accuracy you did not measure.
Built permission-aware rag knowledge assistant using Python, SQLite, Embeddings; evaluated against [baseline] on [N held-out cases], documented [measured result] and [important limitation].Be ready to explain your decisions
- Where exactly are permissions enforced?
- How would you distinguish a retrieval failure from a generation failure?
Answer with a concrete failure from your implementation, the options you considered, your measured evidence and the limitation that remains.
Google documentation and GitHub references
The project scope, milestones and evaluation plans above are original AIJobFever educational material. These official Google resources offer background or implementation examples; they are not finished implementations of this blueprint. No Google endorsement is implied. Review repository licenses, supported platforms and current costs before reusing code or provisioning services.
Our editorial policy