Replayable dataset pipeline for AI

The problem you will solve

An AI team receives daily usage events from a fictional application. Training data must be reproducible even when files arrive late, events repeat or the source schema changes. Build a small batch pipeline that explains rejected rows rather than silently losing them.

Before you start

SQL joins, file processing and primary-key design.

Suggested tech stack

• Python
• SQL
• SQLite
• JSON Schema
• pytest

Data plan

Generate dated JSONL files with stable event IDs, duplicate deliveries, malformed records and late-arriving events. Include a version-two schema fixture. All identifiers should be fictional.

Hardware and cost

Run the initial pipeline on local files and SQLite on CPU. Google Cloud examples are optional architecture references; there is no need to provision a paid streaming cluster for the portfolio version.

Define the input and output

Use this illustrative contract as a starting point, then specify required fields, validation and error responses in your own implementation. It is not a complete API schema.

Input: {"event_id":"e-1","event_time":"2026-10-01T10:00:00Z","schema_version":1,"event":"search"}
Outputs: validated rows, quarantined rows with reason, run manifest with input hashes and counts

Your five build milestones

Finish and test one milestone before extending the system. Keep an experiment log with the decision, the alternative you rejected and the evidence behind the choice.

1. Define event contracts

   Set required fields, timestamps, allowed event names and version rules. Preserve the original input separately so parsing fixes can be replayed without requesting new data.

2. Validate and quarantine

   Route malformed rows to a quarantine table with a stable reason code and source position. Reconcile accepted plus rejected counts to total inputs.

3. Load idempotently

   Choose event_id as the logical deduplication key and document how conflicting payloads are handled. Use transactions so partial runs cannot expose half-written datasets.

4. Handle time and versions

   Distinguish event time from ingestion time. Backfill a late file into the correct partition and migrate version-two records into an explicitly documented canonical form.

5. Publish a dataset manifest

   Record input hashes, code version, output row counts and validation results. Rebuild a dataset from its manifest and compare logical contents after sorting by event ID.

How to evaluate your project

Set your acceptance criteria before looking at the final test results. The checks below are proposed experiments, not measured outcomes or a guarantee that a particular model will pass.

• Reprocess the same files and verify unchanged logical output and no duplicate event IDs.
• Inject a crash halfway through loading and verify atomic recovery.
• Check reconciliation counts and late-event placement; report quarantine reasons rather than hiding rejected rows.

Save your test cases, run command, dependency versions and results together. Compare on identical inputs, retain failures in the denominator and report sample sizes. If a change improves one metric but harms another, explain which trade-off fits your use case.

Failure modes worth demonstrating

• Deduplicating by whole-row equality when the same event can be delivered with different ingestion timestamps.
• Overwriting raw files or silently coercing invalid dates into plausible values.

Optional extension

Add a privacy deletion fixture and show which dataset versions must be rebuilt when an identifier is removed.

Turn the build into portfolio evidence

Your repository should let another person understand the problem and reproduce the core evaluation without needing your private credentials.

• A README with problem, intended user, architecture, setup commands and a small input/output example.
• A permitted sample dataset, its origin or generator, a frozen test split and a single evaluation command.
• A results table comparing your baseline with your final approach, including failures and sample counts.
• A short demo or screenshots, a limitations section and one decision you would change with more time.
• Environment-variable names without secret values, a dependency list and attribution/license notices for any reused code or datasets.

Resume framing

Describe this as a personal or educational project. State what you implemented, how you tested it and what your measured results were. Replace placeholders with your own evidence; do not claim employer impact, production traffic or accuracy you did not measure.

Built replayable dataset pipeline for ai using Python, SQL, SQLite; evaluated against [baseline] on [N held-out cases], documented [measured result] and [important limitation].

Be ready to explain your decisions

• How can a model team reproduce yesterday's dataset?
• What happens when the same event ID has two different payloads?

Answer with a concrete failure from your implementation, the options you considered, your measured evidence and the limitation that remains.

Google documentation and GitHub references

The project scope, milestones and evaluation plans above are original AIJobFever educational material. These official Google resources offer background or implementation examples; they are not finished implementations of this blueprint. No Google endorsement is implied. Review repository licenses, supported platforms and current costs before reusing code or provisioning services.

• Google Cloud: Dataflow GitHub templates ↗