Lead Engineer, AI Quality

Circle seeks a hands-on Lead Engineer to lead its AI Quality engineering team and build the infrastructure used to measure, diagnose, and improve production AI features. The role focuses on evaluation and observability rather than ML research or model training. It is remote, with the listed location of India.

Responsibilities

• Build evaluation infrastructure—including CI/CD pipelines, scorers, and datasets—to assess AI agents, planners, tool-callers, and sub-agents from individual tool calls through multi-turn conversations.
• Trace failures in planning, execution, tool selection, and tool trajectories across product areas such as workflows, site building, and analytics. Prototype fixes or identify priorities for AI Core engineering.
• Develop annotation workflows and AI-generated or simulated conversations to expand evaluation coverage beyond manual labeling.
• Run structured experiments on prompts, models, and the agent harness. Compare new and open-source models with the production baseline, inform model-switching decisions, and improve cost and latency through model swaps, caching, and routing by plan complexity.
• Set technical direction, manage team priorities, remain hands-on in code, and partner with AI Core engineers to verify that product changes improve quality.

Requirements

• 7+ years building and shipping production software, ideally including LLM-powered agents that take actions in a product. Experience with complex, tool-using systems involving multiple tools, planning or orchestration, and sub-agents is sought, along with the ability to explain how shipped work was evaluated.
• Experience building ML/AI evaluation or observability infrastructure, such as evaluation pipelines, scorers, dashboards, or CI/CD for evaluations, and using frameworks such as Braintrust or LangSmith.
• Experience designing datasets, annotation workflows, or labeling pipelines using human annotation, AI-generated data, or simulation. Comfort with Ruby on Rails and Python, or readiness to learn them quickly; Python proficiency is a strong plus. The production system uses Ruby on Rails.
• Ability to experiment in an ambiguous, fast-paced environment, alignment with company values, and spoken, written, and reading English at CEFR C2 / ILR 5.

Compensation and benefits: $170K USD annual cash compensation, paid semi-monthly, plus equity. Circle states that it offers U.S.-benchmarked compensation globally, ongoing equity refresh grants, 35 days of annual paid time off, health, wellbeing, and professional-growth benefits, and twice-yearly company retreats.