Software Engineer III

What matters most

• Evaluation-First Engineering

  The role requires building and owning evaluation infrastructure to measure LLM output quality, including precision, recall, and field-level accuracy. You must prioritize creating test suites and ground truth datasets before deploying any automation, ensuring that AI-driven document extraction meets production-grade standards.

• Robust Python Automation Services

  You are expected to design and maintain production-grade Python services using Clean Architecture principles. This involves building reusable libraries and frameworks that integrate LLM APIs and OCR/IDP workflows, ensuring the system is testable, maintainable, and scalable within AWS environments.

• Formalized Prompt Engineering

  Prompts must be treated as formal specifications rather than experimental scripts. You must demonstrate the ability to design unambiguous, edge-case-aware prompts for complex healthcare documents, while managing versioning, rollback strategies, and lifecycle management for these assets.

• Technical Stakeholder Communication

  You must bridge the gap between technical system constraints and business requirements. This involves translating product needs into system boundaries and clearly communicating model behavior and quality metrics to non-technical stakeholders to ensure alignment on automation goals.

Likely challenges to think through

• Balancing Deterministic Code with Probabilistic AI

  A primary difficulty is integrating non-deterministic LLM outputs into a deterministic, production-grade software pipeline. Prepare by discussing how you implement multi-factor validation and confidence scoring frameworks to catch 'subtly wrong' AI outputs before they impact downstream business processes.

• Managing Complex Healthcare Document Variability

  Healthcare procurement documents like invoices and contracts are highly unstructured and varied. Prepare by explaining your approach to designing classification and extraction pipelines that remain resilient to document format changes while maintaining high accuracy across diverse data sources.

• Scaling Evaluation Infrastructure

  Building an evaluation-first culture requires significant effort in curating ground truth datasets. Prepare to discuss how you would automate the tracking of precision and recall metrics at scale, and how you handle the trade-offs between model latency and extraction accuracy in a production environment.

Topics to prepare

• Evaluation-First Engineering for LLMs

  The role emphasizes building production-grade evaluation pipelines to measure precision, recall, and field-level accuracy. Exercise: Design a test suite for a document extraction task that compares LLM output against a ground-truth dataset, incorporating confidence scoring and validation logic to catch conceptual errors.

• Clean Architecture in AI Pipelines

  The position requires applying software engineering rigor to AI workflows. Exercise: Refactor a monolithic Python script that calls an LLM API into a modular structure using Clean Architecture principles, separating the business logic, data extraction, and external service interfaces to ensure testability.

• Prompt Engineering as Formal Specification

  Prompts are treated as contracts that must be unambiguous and edge-case-aware. Exercise: Draft a structured prompt for extracting data from a complex healthcare invoice, including few-shot examples and chain-of-thought instructions, then document how you would version and rollback this prompt.

Skills to revise

• Python Automation and Async Programming

  The role requires building robust automation services. Exercise: Develop an asynchronous Python service using FastAPI or similar that processes multiple document extraction requests concurrently, ensuring proper error handling and integration with an AWS SQS queue.

• AI Agent Orchestration

  Experience with frameworks like LangChain or LlamaIndex is desired. Exercise: Build a simple agentic workflow using LangChain that retrieves data from a document, validates it against a schema, and performs a secondary check if the confidence score is below a defined threshold.

• AWS Cloud Deployment

  The role involves deploying services on AWS. Exercise: Create a Terraform or AWS CDK script to deploy a Python-based extraction service to AWS Lambda, configured to read from an S3 bucket and write results to a NoSQL database like DynamoDB.

• Design an LLM-based extraction pipeline

  Prepare a technical walkthrough of a system that extracts structured data from unstructured documents. Focus on your approach to prompt engineering (few-shot, chain-of-thought) and how you ensure structured output, referencing your experience with LLM APIs like OpenAI or Anthropic.

• Develop an evaluation-first testing strategy

  Outline how you would build an automated evaluation framework to measure precision, recall, and field-level accuracy for LLM outputs. Explain your process for creating ground truth datasets and validating model behavior against conceptual intent.

• Demonstrate Clean Architecture in Python

  Prepare a code sample or architectural diagram demonstrating how you apply Clean Architecture or Domain-Driven Design (DDD) to a Python service. Highlight how you ensure testability, maintainability, and separation of concerns in your code.

• Prepare AWS deployment and observability examples

  Be ready to discuss your experience deploying Python services on AWS using Lambda, ECS, and SQS. Explain how you monitor these services and handle CI/CD pipelines to ensure production-grade reliability.

• Formulate questions for the IPA team

  Prepare thoughtful questions regarding the team's current document processing challenges, how they balance LLM latency with accuracy requirements, and how they manage the lifecycle of their prompt library in production.