Machine Learning Evaluation: Baselines, Leakage and Error Analysis in Practice

QUICK ANSWER

What you need to know

Learn to design honest ML evaluations, calculate threshold metrics, investigate leakage and compare error slices through a worked example, exercises and a reproducible evaluation project.

A model score is useful only when it answers the deployment question. You need a defensible target, a baseline, independent evaluation data and a clear explanation of the errors that matter. This guide develops those skills without requiring a training framework: you will evaluate fictional scored records and make a threshold decision from explicit evidence.

The code and values are original educational fixtures, not model benchmark results. The tiny sample cannot establish generalization or a suitable threshold for a real service. Its purpose is to make every calculation inspectable before you apply the same reasoning to a larger, appropriately collected dataset.

Before you start

Be comfortable with Python lists and functions, proportions and the distinction between an observed label and a prediction. You do not need a GPU, a cloud account or an installed ML framework. Google's dataset-splitting and classification references provide background; the worked examples and exercises here are separate original material.

Choose a decision with an understandable consequence. For example, a hypothetical review system flags records for a human to inspect. A false alarm consumes review time; a missed positive leaves an issue unreviewed. The relative costs must come from the real workflow, not from the fact that a particular metric is popular.

Learning sequence

1. Define the prediction instant, target and population. List which fields are actually available at that instant.
2. Establish a simple baseline, including the majority-class outcome if appropriate. Evaluate it on the same cases as the candidate.
3. Split data according to deployment: time, person, organization or capture session may matter more than random rows.
4. Choose metrics and a threshold on validation data, then inspect error slices with their sample counts.
5. Freeze the chosen configuration and use a protected final test set once for the stated comparison. Record limitations and plan for monitoring rather than promising permanent accuracy.

Keep model selection separate from the final quality claim. Repeatedly changing the system after inspecting test failures turns the test into another development set. You can learn from those failures, but need new independent evidence before claiming the revised system generalizes.

Worked example

The fixture contains six independent fictional cases with a score and a binary label. A positive prediction is a score greater than or equal to the threshold. The exact boundary rule matters: a score of 0.70 is positive at a threshold of 0.70. The function returns confusion counts and uses None when a proportion has no denominator.

def evaluate(cases, threshold):
    if not 0 <= threshold <= 1:
        raise ValueError("threshold outside [0, 1]")
    tp = fp = fn = tn = 0
    for score, label in cases:
        if type(label) is not int or label not in (0, 1):
            raise ValueError("label must be integer 0 or 1")
        if not 0 <= score <= 1:
            raise ValueError("score outside [0, 1]")
        predicted = score >= threshold
        tp += int(predicted and label == 1)
        fp += int(predicted and label == 0)
        fn += int(not predicted and label == 1)
        tn += int(not predicted and label == 0)
    return {
        "tp": tp, "fp": fp, "fn": fn, "tn": tn,
        "precision": tp / (tp + fp) if tp + fp else None,
        "recall": tp / (tp + fn) if tp + fn else None,
        "review_count": tp + fp,
    }

cases = [(0.95, 1), (0.80, 0), (0.70, 1),
         (0.55, 0), (0.40, 1), (0.10, 0)]
at_half = evaluate(cases, 0.50)
at_high = evaluate(cases, 0.90)
assert (at_half["tp"], at_half["fp"], at_half["fn"], at_half["tn"]) == (2, 2, 1, 1)
assert at_half["precision"] == 0.5
assert at_half["recall"] == 2 / 3
assert at_half["review_count"] == 4
assert at_high["precision"] == 1.0
assert at_high["recall"] == 1 / 3
assert evaluate([], 0.5)["precision"] is None
print("0.50: review 4, precision 0.5, recall 2/3")
print("0.90: review 1, precision 1.0, recall 1/3")

At 0.50, the reviewer receives four cases and finds two positives. At 0.90, the reviewer receives one case and it is positive, but two of the three actual positives are missed. The higher precision is therefore not a universal improvement. You need the review capacity and consequence of missed issues to choose a sensible operating point.

An undefined precision is not identical to zero precision. The former can mean no cases were predicted positive; the latter means positive predictions existed but none were correct. Preserve that distinction in reports and avoid presenting an empty evaluation as successful performance.

Exercises and solutions

Exercise 1: Evaluate the middle threshold

Using the same fixture, calculate confusion counts, review count, precision and recall at 0.70 before running the code.

Solution: Scores 0.95, 0.80 and 0.70 are positive predictions: two true positives and one false positive. There is one false negative and two true negatives. Review count is three, precision is 2/3 and recall is 2/3. This is fixture arithmetic, not evidence that 0.70 is a good threshold for another dataset.

Exercise 2: Find the leakage

A cancellation predictor uses an account's final closure reason. Training and test rows are randomly separated and the score looks excellent. Is the random split enough to validate deployment?

Solution: No. If closure reason is recorded after the prediction instant, it contains future information unavailable when the model must act. Remove it or reconstruct the information actually available then. Also inspect repeated entities and time dependence. An independent-looking row partition cannot repair a target definition that already leaks the outcome.

Exercise 3: Interpret a small error slice

One language group has two errors among three examples. Another has twenty errors among two hundred examples. Can you confidently rank their long-term performance using the percentages alone?

Solution: Report both counts and denominators and investigate the small group, but do not treat three cases as a stable population estimate. Review labels, sampling and conditions, then gather appropriate evidence. The larger observed proportion is a useful diagnostic signal; it is not a sufficiently precise standalone claim about the whole language population.

Mini-project

Build an evaluation report for a fictional binary review task. Store case IDs, timestamps, entity IDs, scores, labels and one relevant slice. Document the prediction instant and demonstrate a split that respects it. You may use fixed synthetic scores initially; do not claim to have trained a model when you have not.

Deliver a baseline comparison, a threshold table, error examples, slice counts and a short decision note. The note should say what evidence justifies the selected threshold, which failure matters most and what remains unknown. Include a held-out test policy and record which cases were used to make decisions.

Completion criteria: confusion counts reconcile to the fixture size; boundary cases are tested; empty denominators are explicit; duplicate entities are addressed; the candidate and baseline use the same evaluation cases; and every reported measurement is labelled synthetic when it comes from your fixture. Include at least one reason you would decline deployment despite an improved aggregate score.

Common mistakes

• Reporting accuracy alone for a task where a trivial negative prediction already performs well.
• Fitting preprocessing on evaluation records before splitting.
• Selecting a threshold on the final test set and calling the result unbiased.
• Comparing different cohorts or time windows without stating the mismatch.
• Reporting a tiny slice's percentage without its denominator.
• Treating a probability-like score as calibrated confidence without checking it.

Self-assessment

Can you calculate a confusion matrix by hand, explain a threshold's operational consequence and identify information unavailable at prediction time? Can you describe what the test set does and does not establish? If you cannot explain an apparent improvement without naming the model, revisit the experimental design before tuning architecture or collecting more parameters.

Connect this skill to roles and jobs

Read the Machine Learning Engineer career guide, practise ML Engineer interview questions and explore machine-learning jobs. Use a real listing to identify which evaluation responsibilities it actually includes. Return to the AI Skills overview for the broader stack.

HOW THIS WAS PREPARED

Methodology

Editorial synthesis of cited sources

AI-assisted original educational material. Explanations, code, fictional fixtures, exercise solutions and mini-project acceptance criteria were prepared for this site, not reproduced from the references. Official Google references support background concepts, not an endorsement or validation of these projects. Runnable Python examples use the standard library; the SQL fixture is checked with SQLite and is not a BigQuery deployment tutorial. Samples demonstrate bounded concepts, not production-ready systems. Measurements and scenario values are synthetic, not benchmark results. Mini-projects are suggested exercises, not claims of completed deployments. No paid model calls are needed to run the examples. Related jobs use a skill or specialization filter and may include adjacent roles. References checked 17 October 2026. No hiring guarantees or unmeasured demand rankings are implied.

Read our editorial and research policy →

EVIDENCE

Sources

1. Dividing datasets

   Google for Developers · Checked 17 October 2026

2. Classification metrics

   Google for Developers · Checked 17 October 2026