Applied AI Engineer

What matters most

• Build an end-to-end voice evaluation harness

  Posting: The role centers on scoring synthetic and real calls across STT, LLM reasoning, tool use, TTS, and full-call outcomes; the day-30 target is a trusted end-to-end metric. Prepare: Outline how you would define that metric and inspect failures at each layer.

• Turn evaluations into release safeguards

  Posting: By day 60, evaluations should run on updates and block merges that regress on known cases. Scheduled red-team tests should cover jailbreaks, hallucinated quotes, and compliance failures. Prepare: Describe how results would become actionable CI signals.

• Build a production-failure feedback loop

  Posting: The role includes reviewing calls, mining hard cases from production traffic, growing the evaluation set, and feeding fixes into prompt optimization; a live loop for one failure pattern is a day-90 target. Prepare: Trace one failure from discovery through evaluation, fix, and retest.

• Make defensible model comparisons quickly

  Posting: The role benchmarks new open and closed models against the production stack, with a day-90 goal of a numbers-backed ship decision within a week. Prepare: Define a repeatable comparison that includes full-call outcomes, not just model-level scores.

• Improve real-time agent and audio quality

  Posting: Duties include prompts, tool use, conversation flow, recovery, guardrails, STT/TTS provider comparisons, A/B experiments, and TTS latency-versus-fidelity decisions. Prepare: Connect an audio or behavior change to a measurable conversation outcome.

Likely challenges to think through

• Making a metric trustworthy

  Likely challenge, not a confirmed problem: A single full-call score may conceal transcription errors, faulty tool use, or missed disclosures, while LLM-as-judge pipelines have known limitations. Prepare to explain how human-reviewed call failures and layer-level checks would validate the metric.

• Blocking regressions without blocking useful changes

  Likely challenge, not a confirmed problem: CI gates must catch consequential failures without treating variable conversation scores as definitive. Prepare to discuss stable known-case tests, thresholds, and how to investigate a failed gate before changing it.

• Scaling hard-case mining without diluting the eval set

  Likely challenge, not a confirmed problem: Automated mining reduces manual triage but may add repetitive or poorly labeled calls. Prepare to describe sampling, failure categorization, and periodic review before cases enter a regression suite.

• Balancing speed, voice quality, and decision rigor

  Likely challenge, not a confirmed problem: A one-week model comparison and real-time voice constraints leave limited room for exhaustive testing; lower latency may cost speech fidelity or conversation quality. Prepare to compare candidates on representative calls, quality failures, and latency rather than one aggregate score.

Topics to prepare

• End-to-end evaluation and judge reliability

  The role scores transcription, LLM reasoning, tool use, TTS, and full-call outcomes; LLM-as-judge limitations are a must-have. Revision exercise: sketch a scoring rubric for a synthetic insurance-agency call, then explain how human review would catch judge errors and validate a trusted end-to-end metric.

• Real-time agent behavior

  The posting names prompts, function calling, structured outputs, multi-turn state, error recovery, and guardrails. Revision exercise: diagram a lead-qualification call with a tool failure and a corrected recovery path; explain which transitions and outputs an evaluation should inspect.

• Speech pipeline tradeoffs

  Practical ASR/STT, TTS, and audio knowledge are required, including sample rates, codecs, noise, and alignment. Revision exercise: trace how noisy or misaligned speech could affect transcription, agent response, and synthesized voice; propose separate measures for accuracy, prosody, and latency versus fidelity.

• Regression gates and model comparisons

  The role calls for benchmarks against the production stack and evaluations that block regressing merges. Revision exercise: outline a fair comparison using the same call cases and scoring rules, including how you would investigate a failing gate before recommending a model switch.

• Failure taxonomies and improvement loops

  The posting specifies red-team coverage for jailbreaks, hallucinated quotes, and compliance failures, plus mining hard cases from production calls. Revision exercise: define one test case and expected failure signal for each class, then trace how a confirmed failure becomes a reusable evaluation case. Do not assume a particular insurance disclosure rule.

• Measuring prompt-optimization lift

  A day-90 milestone calls for measurable improvement from DSPy- or GEPA-style optimization on a production voice flow. Revision exercise: describe a baseline, a held-out call set, an end-to-end metric, and a regression check that would distinguish improvement from overfitting.

Skills to revise

• Build a Python voice-evaluation harness

  The posting prioritizes an evaluation harness over audio work alone. Practice implementing a small Python runner for synthetic call records that stores per-layer scores for STT, LLM behavior, tool use, TTS, and full-call outcome, then produces a readable regression report.

• Audit an LLM-as-judge pipeline

  Hands-on judge pipelines and knowledge of their failure modes are required. Practice writing a scoring prompt for agent responses, comparing its decisions with hand-labeled cases, and recording disagreement patterns rather than treating judge scores as ground truth.

• Analyze evaluation data with the stated ML stack

  Strong Python and a working stack of PyTorch, Hugging Face, pandas, and scikit-learn are listed as must-haves. Practice using pandas to group call failures by type and scikit-learn to summarize a labeled error set; rehearse where PyTorch and Hugging Face would fit if an open model were benchmarked.

• Run a speech-quality comparison

  The role compares and tunes STT and TTS systems. Practice evaluating two configurations on the same sample calls, inspecting transcription errors and TTS edge cases alongside latency, and writing an A/B recommendation with its limitations. Whisper and ElevenLabs are examples named in the posting, not mandatory choices for this exercise.

• Turn hard cases into regression tests

  The posting calls for automated production hard-case mining, scheduled red-team tests, and CI gates. Practice writing a small script that selects likely failures from tagged call records, deduplicates them into test cases, and fails an update when a known case regresses.

• Benchmark and improve one agent flow

  The milestones call for a defensible new-model comparison and prompt optimization with measurable lift. Practice running the same multi-turn cases through two model or prompt variants, checking tool calls and structured outputs, and reporting outcome changes and failure cases on held-out calls.

• Select a production project walkthrough

  The role requires shipped ML systems and hands-on LLM-agent evaluation. If you have a relevant project, prepare a one-page account of your Python and ML stack, agent design, evaluation choices, results, and your contribution; otherwise, prepare a clearly labeled design example.

• Sketch an end-to-end voice evaluation harness

  Infer wants synthetic and real calls scored across STT, LLM reasoning, tool use, TTS, and call outcomes. Prepare a diagram, a proposed end-to-end metric, and a plan to validate it against human-reviewed calls, including where LLM-as-judge scores could mislead.

• Walk through a real-time agent failure

  Prepare a hypothetical insurance-agency call showing multi-turn state, a tool call, structured output, and recovery from an error. Explain how prompts and guardrails would address a hallucinated quote or missed disclosure without assuming Infer’s exact policy.

• Outline an audio comparison experiment

  The role compares STT and TTS providers and runs A/B tests. Draft a test plan covering noisy audio, sample rates, codecs, alignment, transcription accuracy, voice quality, prosody, and latency-versus-fidelity tradeoffs; identify what evidence would support a decision.

• Build a model-benchmark template

  Infer expects defensible comparisons with its production stack, including an open-source model. Prepare a scorecard for quality, tool behavior, failure rates, latency, and cost, with a recommendation rule; leave results blank rather than inventing numbers.

• Design a red-team and CI gate

  Sketch test cases for the three stated failure classes: jailbreaks, hallucinated quotes, and compliance failures. Show how scheduled tests and a known-case regression set could produce actionable results and block a regressing merge.

• Map a failure-to-improvement loop

  Prepare a proposed path from reviewed production calls to hard-case mining, dataset updates, retesting, and a measurable DSPy- or GEPA-style prompt experiment. Specify how you would check that improvement on one failure pattern does not worsen other call outcomes.

• Bring questions about evaluation ownership

  Ask which call outcomes the team trusts today, how customer call reviews inform labels, what data and backend interfaces are available for CI, and how the day-30, day-60, and day-90 milestones will be measured.