Visual product search with text queries

The problem you will solve

A user remembers a product's appearance but not its name. Build a search prototype that retrieves catalogue images for phrases such as a small blue cup. Unlike image-quality checking, this project tests whether language and image representations retrieve the right objects, including cases with similar colours but different functions.

Before you start

Vector similarity, image preprocessing and ranked retrieval metrics.

Suggested tech stack

• Python
• NumPy
• Pillow
• Multimodal embeddings
• FastAPI

Data plan

Photograph 40–60 non-personal household objects that you own or have permission to use. Record image rights, stable IDs and descriptions. Write independent query descriptions and label acceptable matches, including searches with no valid match.

Hardware and cost

Image storage, keyword ranking and vector comparison work on CPU for a small catalogue. A hosted multimodal embedding service may charge for image and text processing. Cache only permitted vectors and keep credentials on the server, not in browser code.

Define the input and output

Use this illustrative contract as a starting point, then specify required fields, validation and error responses in your own implementation. It is not a complete API schema.

Input: {"query":"small blue ceramic cup","top_k":5}
Output: {"matches":[{"image_id":"cup-07","score":0.72}],"embedding_version":"recorded-version"}
Similarity scores are not probabilities.

Your five build milestones

Finish and test one milestone before extending the system. Keep an experiment log with the decision, the alternative you rejected and the evidence behind the choice.

1. Build the catalogue

   Assign stable image IDs, normalize orientation and record usage permissions. Include visually confusing pairs and keep variants of the same photograph grouped in evaluation.

2. Make a lexical baseline

   Search the written catalogue descriptions using keyword overlap. Save the ranked IDs for every labeled query before introducing embeddings so improvement can be tested fairly.

3. Index images and queries

   Choose an embedding model supporting text and images in the same space. Record model and preprocessing versions, then compute vectors once per catalogue image and for each query.

4. Rank and handle uncertainty

   Use a similarity method compatible with the chosen model and return a bounded ranked list. Tune any no-match threshold on development queries, not on the final test set.

5. Explain retrieval errors

   Display permitted thumbnails and a failure gallery for colour, shape and function confusion. Document query latency, indexing effort and where the keyword baseline remains better.

How to evaluate your project

Set your acceptance criteria before looking at the final test results. The checks below are proposed experiments, not measured outcomes or a guarantee that a particular model will pass.

• Measure recall@5 and mean reciprocal rank against labeled acceptable matches, using the same queries for both approaches.
• Report no-match false positives separately; do not interpret raw similarity as confidence in correctness.
• Inspect performance on object function, colour and composition queries, with sample counts per category.

Save your test cases, run command, dependency versions and results together. Compare on identical inputs, retain failures in the denominator and report sample sizes. If a change improves one metric but harms another, explain which trade-off fits your use case.

Failure modes worth demonstrating

• Comparing text and image vectors produced by incompatible embedding models.
• Using the exact catalogue caption as every test query, making retrieval look unrealistically easy.

Optional extension

Support image-to-image search and compare its behaviour with text-to-image search on the same object groups.

Turn the build into portfolio evidence

Your repository should let another person understand the problem and reproduce the core evaluation without needing your private credentials.

• A README with problem, intended user, architecture, setup commands and a small input/output example.
• A permitted sample dataset, its origin or generator, a frozen test split and a single evaluation command.
• A results table comparing your baseline with your final approach, including failures and sample counts.
• A short demo or screenshots, a limitations section and one decision you would change with more time.
• Environment-variable names without secret values, a dependency list and attribution/license notices for any reused code or datasets.

Resume framing

Describe this as a personal or educational project. State what you implemented, how you tested it and what your measured results were. Replace placeholders with your own evidence; do not claim employer impact, production traffic or accuracy you did not measure.

Built visual product search with text queries using Python, NumPy, Pillow; evaluated against [baseline] on [N held-out cases], documented [measured result] and [important limitation].

Be ready to explain your decisions

• Why must image and text vectors share a compatible representation?
• How did you label queries with several acceptable images?

Answer with a concrete failure from your implementation, the options you considered, your measured evidence and the limitation that remains.

Google documentation and GitHub references

The project scope, milestones and evaluation plans above are original AIJobFever educational material. These official Google resources offer background or implementation examples; they are not finished implementations of this blueprint. No Google endorsement is implied. Review repository licenses, supported platforms and current costs before reusing code or provisioning services.

• Google: text and multimodal embeddings ↗