Fine-tuned support ticket classifier

The problem you will solve

A fictional software team routes support tickets to a small set of queues. Investigate whether adapting an open-weight model improves category consistency enough to justify training cost. Fine-tuning is a hypothesis to test, not an automatic upgrade over a simpler baseline or a general model with examples.

Before you start

Train/validation/test separation, language-model inference, classification metrics and GPU runtime basics.

Suggested tech stack

• Python
• Gemma
• Keras
• LoRA
• pandas

Data plan

Author a small, clearly synthetic ticket dataset across billing, export, login and product-feedback categories, with ambiguous cases labeled review. Group paraphrases from the same source ticket into one partition and keep the final test set untouched.

Hardware and cost

This is the GPU/cloud track, not a promise of CPU-laptop training. Google's referenced LoRA tutorial uses a compatible GPU runtime. Check model access terms, memory needs and current runtime costs; cap sequence length and run size, and keep the untuned comparison available if hardware is insufficient.

Define the input and output

Use this illustrative contract as a starting point, then specify required fields, validation and error responses in your own implementation. It is not a complete API schema.

Input: {"text":"My demo export fails after selecting CSV."}
Output: {"category":"export","requires_review":false}
Experiment: base_model_version, adapter_version, split_hash, prompt_version, seed, runtime

Your five build milestones

Finish and test one milestone before extending the system. Keep an experiment log with the decision, the alternative you rejected and the evidence behind the choice.

1. Freeze labels and partitions

   Define mutually understood queue labels and an ambiguity policy. Deduplicate grouped source tickets before splitting, and document the synthetic data-generation process and its limitations.

2. Measure an untuned baseline

   Run a fixed prompt with the allowed categories and a few development examples. Record schema failures, per-class errors and latency before spending resources on training.

3. Run bounded LoRA adaptation

   Use a supported Gemma/Keras configuration and parameter-efficient adaptation after checking model terms. Log trainable parameter count, sequence length, epochs and actual GPU time; never embed credentials in notebooks.

4. Compare held-out decisions

   Evaluate the adapted and untuned versions on the same untouched tickets using the same label rules. Count invalid outputs as failures and inspect whether rare classes were harmed.

5. Package a model decision card

   Record base-model and adapter versions together, publish permitted evaluation examples and state whether adaptation was worth the resources. Do not distribute weights or data without checking their terms.

How to evaluate your project

Set your acceptance criteria before looking at the final test results. The checks below are proposed experiments, not measured outcomes or a guarantee that a particular model will pass.

• Compare macro-F1, per-class precision/recall and invalid-label rate, with class support counts.
• Test ambiguous and out-of-domain tickets separately; a confident category on unknown input may be a failure.
• Report training runtime and inference latency alongside quality, and acknowledge that synthetic-only results do not prove real support-routing performance.

Save your test cases, run command, dependency versions and results together. Compare on identical inputs, retain failures in the denominator and report sample sizes. If a change improves one metric but harms another, explain which trade-off fits your use case.

Failure modes worth demonstrating

• Training on paraphrases of held-out tickets or repeatedly tuning on the final test results.
• Saving an adapter without its compatible base-model version or claiming LoRA eliminates GPU memory requirements.

Optional extension

Add a human-review threshold and compare how much review effort each configuration needs at the same error tolerance.

Turn the build into portfolio evidence

Your repository should let another person understand the problem and reproduce the core evaluation without needing your private credentials.

• A README with problem, intended user, architecture, setup commands and a small input/output example.
• A permitted sample dataset, its origin or generator, a frozen test split and a single evaluation command.
• A results table comparing your baseline with your final approach, including failures and sample counts.
• A short demo or screenshots, a limitations section and one decision you would change with more time.
• Environment-variable names without secret values, a dependency list and attribution/license notices for any reused code or datasets.

Resume framing

Describe this as a personal or educational project. State what you implemented, how you tested it and what your measured results were. Replace placeholders with your own evidence; do not claim employer impact, production traffic or accuracy you did not measure.

Built fine-tuned support ticket classifier using Python, Gemma, Keras; evaluated against [baseline] on [N held-out cases], documented [measured result] and [important limitation].

Be ready to explain your decisions

• What evidence would make you reject fine-tuning?
• How did you prevent paraphrase leakage and track the base model plus adapter?

Answer with a concrete failure from your implementation, the options you considered, your measured evidence and the limitation that remains.

Google documentation and GitHub references

The project scope, milestones and evaluation plans above are original AIJobFever educational material. These official Google resources offer background or implementation examples; they are not finished implementations of this blueprint. No Google endorsement is implied. Review repository licenses, supported platforms and current costs before reusing code or provisioning services.

• Google: fine-tune Gemma in Keras using LoRA ↗