Semantic duplicate detection review queue

The problem you will solve

A fictional resource catalogue contains reposted entries with changed wording, along with genuinely different entries on the same subject. Build candidate matching and a review queue, not an irreversible automatic merge. Unlike event-ID deduplication, semantic duplication requires evidence about whether two records describe the same underlying item.

Before you start

Text normalization, vector similarity, pairwise labels and database identifiers.

Suggested tech stack

• Python
• Embeddings
• NumPy
• SQLite
• FastAPI

Data plan

Author 80 fictional catalogue records with known duplicate groups and difficult non-duplicate pairs. Include changed dates, similar titles and shared organizations. Split by underlying group so paraphrases cannot cross development and test partitions.

Hardware and cost

Exact hashing and small-catalogue vector comparisons run on CPU. Embedding generation may use a fitting local model or a hosted service with charges. Keep text and vector versions recorded; use a small catalogue before considering a vector database.

Define the input and output

Use this illustrative contract as a starting point, then specify required fields, validation and error responses in your own implementation. It is not a complete API schema.

Input: {"record_id":"r-22","text":"..."}
Output: {"candidates":[{"record_id":"r-08","similarity":0.86,"status":"needs_review","evidence":["same canonical source"]}]}
Review action: link_as_duplicate / keep_distinct / unresolved

Your five build milestones

Finish and test one milestone before extending the system. Keep an experiment log with the decision, the alternative you rejected and the evidence behind the choice.

1. Define duplicate identity

   Write a rubric for the same underlying item versus a merely related item. Establish which structured conflicts, such as incompatible dates or source IDs, require manual inspection.

2. Build exact matching first

   Normalize harmless formatting and hash the resulting text. Preserve original records and measure which known duplicate groups this cheap baseline finds before adding semantic representations.

3. Generate candidate pairs

   Embed permitted text and retrieve a bounded candidate list per record. Record vector versions and avoid evaluating every random pair as an easy negative that inflates accuracy.

4. Tune review decisions

   Choose development-set thresholds for review, not deletion. Display matching evidence and conflicts, letting a reviewer keep distinct records even when similarity is high.

5. Retain reversible provenance

   Store original IDs, review outcome and reason without overwriting source records. Demonstrate undoing an incorrect grouping and explain why transitive pair matches can create a bad cluster.

How to evaluate your project

Set your acceptance criteria before looking at the final test results. The checks below are proposed experiments, not measured outcomes or a guarantee that a particular model will pass.

• Measure candidate recall@5 before assessing the reviewer threshold, then pair precision/recall on labeled hard cases.
• Report false merges and missed duplicates separately; do not hide rare duplicate prevalence behind overall accuracy.
• Test a three-record chain where A resembles B and B resembles C but A and C represent different items.

Save your test cases, run command, dependency versions and results together. Compare on identical inputs, retain failures in the denominator and report sample sizes. If a change improves one metric but harms another, explain which trade-off fits your use case.

Failure modes worth demonstrating

• Interpreting embedding similarity as proof of identity or deleting a source record automatically.
• Evaluating random easy negatives while excluding near-identical but distinct records.

Optional extension

Add a cluster-review view that highlights structured conflicts and supports splitting an incorrectly linked group.

Turn the build into portfolio evidence

Your repository should let another person understand the problem and reproduce the core evaluation without needing your private credentials.

• A README with problem, intended user, architecture, setup commands and a small input/output example.
• A permitted sample dataset, its origin or generator, a frozen test split and a single evaluation command.
• A results table comparing your baseline with your final approach, including failures and sample counts.
• A short demo or screenshots, a limitations section and one decision you would change with more time.
• Environment-variable names without secret values, a dependency list and attribution/license notices for any reused code or datasets.

Resume framing

Describe this as a personal or educational project. State what you implemented, how you tested it and what your measured results were. Replace placeholders with your own evidence; do not claim employer impact, production traffic or accuracy you did not measure.

Built semantic duplicate detection review queue using Python, Embeddings, NumPy; evaluated against [baseline] on [N held-out cases], documented [measured result] and [important limitation].

Be ready to explain your decisions

• What makes a duplicate different from a related record?
• Why can individually plausible pair matches produce an incorrect cluster?

Answer with a concrete failure from your implementation, the options you considered, your measured evidence and the limitation that remains.

Google documentation and GitHub references

The project scope, milestones and evaluation plans above are original AIJobFever educational material. These official Google resources offer background or implementation examples; they are not finished implementations of this blueprint. No Google endorsement is implied. Review repository licenses, supported platforms and current costs before reusing code or provisioning services.

• Google: text and multimodal embeddings ↗