Experiment readout and decision notebook

The problem you will solve

A product team tests a revised onboarding flow. Analyse whether the evidence supports rollout while checking that assignment and measurement are trustworthy. The project demonstrates decision-making under uncertainty, not a claim that a real business achieved an uplift.

Before you start

Averages, proportions, SQL grouping and basic probability.

Suggested tech stack

• Python
• pandas
• NumPy
• SQL
• Jupyter

Data plan

Generate fictional user-level assignments and events with a reproducible seed. Include duplicate events, missing outcomes and a segment with a different baseline. Never use identifiable customer events.

Hardware and cost

A few thousand synthetic rows are enough for a CPU notebook. No model API or GPU is required; the AI-career value is rigorous experimentation supporting AI product decisions.

Define the input and output

Use this illustrative contract as a starting point, then specify required fields, validation and error responses in your own implementation. It is not a complete API schema.

Input: user_id, assigned_group, assignment_time, activation_event, support_events
Output: arm counts, activation rates, estimated difference, interval, quality checks, rollout recommendation

Your five build milestones

Finish and test one milestone before extending the system. Keep an experiment log with the decision, the alternative you rejected and the evidence behind the choice.

1. Write the decision first

   Define activation within seven days as the primary outcome, user as the unit, and support contacts as a guardrail. Record a practical effect threshold before inspecting simulated results.

2. Validate assignment

   Compare arm counts with the intended allocation and inspect duplicate assignments. Build a single user-level analysis table without accidentally counting frequent users multiple times.

3. Estimate uncertainty

   Compute activation rates and their difference. Bootstrap users with a fixed seed for an interval, documenting assumptions and why individual events are not independent units.

4. Inspect robustness

   Report missing outcomes by arm and a prespecified segment view. Explain that exploratory slices are hypothesis-generating rather than definitive discoveries.

5. Deliver a decision memo

   Provide a one-page readout with estimates, uncertainty, guardrails and conditions for further testing. Include the data-generation script and a notebook that runs from a clean environment.

How to evaluate your project

Set your acceptance criteria before looking at the final test results. The checks below are proposed experiments, not measured outcomes or a guarantee that a particular model will pass.

• Test the analysis on a simulated zero-effect dataset and a deliberately unbalanced assignment.
• Check that duplicated event rows do not change user-level activation counts.
• Show sensitivity to missing outcomes and explain why a statistically detectable change might be too small to matter.

Save your test cases, run command, dependency versions and results together. Compare on identical inputs, retain failures in the denominator and report sample sizes. If a change improves one metric but harms another, explain which trade-off fits your use case.

Failure modes worth demonstrating

• Repeatedly inspecting results and stopping only when they look positive.
• Claiming causation from a non-random comparison or reporting only favourable segments.

Optional extension

Add a power simulation over multiple sample sizes and document its assumed effect, rather than claiming one universal sample-size requirement.

Turn the build into portfolio evidence

Your repository should let another person understand the problem and reproduce the core evaluation without needing your private credentials.

• A README with problem, intended user, architecture, setup commands and a small input/output example.
• A permitted sample dataset, its origin or generator, a frozen test split and a single evaluation command.
• A results table comparing your baseline with your final approach, including failures and sample counts.
• A short demo or screenshots, a limitations section and one decision you would change with more time.
• Environment-variable names without secret values, a dependency list and attribution/license notices for any reused code or datasets.

Resume framing

Describe this as a personal or educational project. State what you implemented, how you tested it and what your measured results were. Replace placeholders with your own evidence; do not claim employer impact, production traffic or accuracy you did not measure.

Built experiment readout and decision notebook using Python, pandas, NumPy; evaluated against [baseline] on [N held-out cases], documented [measured result] and [important limitation].

Be ready to explain your decisions

• What would invalidate your experiment readout?
• How would you communicate an inconclusive result?

Answer with a concrete failure from your implementation, the options you considered, your measured evidence and the limitation that remains.

Google documentation and GitHub references

The project scope, milestones and evaluation plans above are original AIJobFever educational material. These official Google resources offer background or implementation examples; they are not finished implementations of this blueprint. No Google endorsement is implied. Review repository licenses, supported platforms and current costs before reusing code or provisioning services.

• Google: Machine Learning Crash Course ↗