Senior Engineer, Applied AI

What matters most

• Own LLM features through production

  Role: Design and ship features end-to-end, building orchestration, tool calling, retrieval, agent workflows, and structured outputs. Preparation: Be ready to explain architecture choices and how a feature moved from prototype to reliable use.

• Make quality measurable

  Role: Build regression suites, LLM-as-a-judge evaluations, synthetic datasets, human review workflows, and production observability for accuracy, latency, cost, hallucinations, and model behavior. Preparation: Organize an example around what was measured, what failed, and what changed.

• Build safety and reliability guardrails

  Role: Ensure AI systems meet quality, compliance, and operational requirements in healthcare workflows. Preparation: Be ready to discuss how guardrails, testing, and monitoring work together; the posting does not specify particular controls.

• Learn workflows and iterate on outcomes

  Role: Work directly with EHR operators, Patient Advocates, and clinical teams; prototype, ship, measure outcomes, and iterate. Preparation: Explain how operational observations can become a scoped technical solution and how real-world usage would inform changes.

• Exercise senior engineering judgment

  Role: Make build-versus-buy decisions across AI tools and raise standards through system design, code reviews, and mentorship. Preparation: Structure examples around trade-offs, decision criteria, and the effect on the team.

Likely challenges to think through

• Shipping quickly without weakening safeguards

  Potential challenge, not a reported problem: Early prototypes and tight feedback loops may pull against the reliability needed for patient and clinical operations. Prepare to explain how you would stage release, define quality thresholds, and respond when a guardrail fails.

• Reconciling evaluation signals with real use

  Potential challenge, not a reported problem: Synthetic data and model-graded tests can catch regressions yet miss failures that operators encounter, while human review takes time. Prepare to discuss when to use each signal and how production observations would update an evaluation suite.

• Choosing tools for an evolving function

  Potential challenge, not a reported problem: A new team with an unsettled reporting structure may need decisions before ownership boundaries are fully clear; buying tools speeds delivery but can constrain future changes. Prepare a decision framework covering fit, operational cost, reversibility, and who needs to be involved.

Topics to prepare

• LLM application architecture

  The posting calls for prompt orchestration, tool calling, retrieval, agent workflows, and structured outputs. As a revision exercise, sketch a feature from user request to backend response and explain where each component helps, where it adds complexity, and how failures are handled.

• Evaluation strategy

  Ro specifies regression suites, LLM-as-a-judge evaluations, synthetic datasets, human review, and quality metrics. Design a hypothetical evaluation plan that combines these methods, identifies what each can miss, and sets criteria for detecting regressions before release.

• Production reliability and safety

  The role measures accuracy, latency, cost, hallucinations, and model behavior while meeting quality, compliance, and operational requirements. Map likely failure modes for a hypothetical clinical-operations workflow to guardrails, monitoring signals, and escalation to human review; this is a preparation scenario, not a stated Ro process.

• Tool selection and build-versus-buy

  The engineer will assess orchestration frameworks, evaluation platforms, vector databases, and model providers. Draft a comparison rubric covering integration effort, quality, reliability, latency, cost, and operational control, then explain a build-versus-buy choice. Named products in the posting are bonus experience, not requirements.

• Workflow discovery and outcome measurement

  The posting emphasizes working with EHR operators, Patient Advocates, and clinical teams, then measuring outcomes and iterating. Practice turning a hypothetical operator pain point into a scoped prototype, questions for operators, and measurable user and business outcomes; do not assume a particular Ro workflow.

Skills to revise

• Python backend integration

  Strong Python, backend services, and AI API integration are required. Build a small service endpoint that calls a model API, validates its response, handles timeouts and errors, and records latency and cost; explain how you would take it from prototype to production.

• Prompt and tool workflow implementation

  The role builds orchestration layers, prompts, tool calls, agent workflows, and structured outputs. Implement a short workflow against mock tools, validate the output format, and test invalid tool results and ambiguous requests.

• Retrieval pipeline prototyping

  Retrieval, embeddings, and RAG are listed responsibilities; vector databases are also a tool-selection area. Using fictional documents, prototype retrieval-backed answers and inspect cases where retrieved material is missing, irrelevant, or insufficient. Treat any particular framework or database as optional.

• Evaluation suite construction

  The posting calls for evaluation infrastructure. Create a small synthetic test set, run regression checks on outputs, compare an LLM-as-a-judge score with a manual review, and document disagreements and release-blocking failures.

• Observability and guardrail checks

  Ro asks for production-behavior observability and safety guardrails. Instrument a toy feature to report quality, latency, cost, and suspected hallucinations; add a fallback for unsupported answers and show how a failure would be investigated.

• Technical review and AI-assisted development

  The role includes code reviews, mentorship, system design, and comfort using AI-assisted development tools. Review a sample LLM-service change for reliability and test coverage, write constructive review comments, and demonstrate how you would verify rather than blindly accept AI-assisted code.

• Map your evidence to the role

  Prepare a one-page inventory of relevant production software, Python backend, LLM application, AI API, and AI-assisted development work. Ro asks for production experience or equivalent depth through substantial side projects or open-source work; label each example accurately.

• Rehearse an end-to-end feature walkthrough

  Choose a real example, if available, and explain the user problem, prototype, architecture, launch, and iteration. Bring a diagram and the quality, reliability, latency, cost, and business-impact measures you actually used; do not invent results.

• Practice a healthcare-workflow design exercise

  Sketch a hypothetical LLM tool for an EHR operator or Patient Advocate. State assumptions, questions for clinical operators, backend and orchestration choices, retrieval or tool-calling boundaries, structured outputs, and a path from prototype to production.

• Draft an evaluation and observability plan

  For the hypothetical design, produce a small test plan covering regression cases, synthetic data, human review, and possible LLM-as-a-judge checks. Specify how you would monitor accuracy, hallucinations, latency, cost, and production behavior.

• Define safety and reliability checks

  Write a failure-mode table for the design, with proposed guardrails, escalation to human review, and checks before release. Treat specific compliance obligations as questions to confirm with Ro rather than assumed requirements.

• Compare build-versus-buy options

  Prepare a short decision memo comparing plausible orchestration, evaluation, vector database, or model-provider options against quality, operational complexity, and cost. Explain what evidence would change your choice; the named frameworks and platforms are bonus experience, not mandatory tools.

• Prepare leadership examples

  Outline specific instances, if available, of system-design decisions, code reviews, mentorship, or resolving ambiguous requirements. Note your contribution and what changed, without attributing team outcomes solely to yourself.

• Bring questions about the new team

  Ask how clinical operators participate in defining workflows and reviewing outcomes, which AI quality measures matter most, and how ownership will be divided with the Senior AI Lead and Engineering Manager. Ro says the new function’s reporting structure is not yet fixed.