Applied Scientist, Foundational AI

The Artificial General Intelligence (AGI) team seeks an Applied Scientist to improve the quality of training and evaluation data for Amazon Nova models, LLM products, and multimodal systems. The role combines data-quality strategy, auditing, and automated assessment in collaboration with scientists, engineers, domain experts, and vendor teams.

Responsibilities

• Develop quality strategies and auditing frameworks for data collection workflows, including standard operating procedures, quality metrics, and sampling methods that support benchmark performance.
• Perform manual audits and meta-audits of auditor performance, provide targeted coaching, and communicate quality feedback to stakeholders.
• Develop and maintain LLM-as-a-Judge systems: design judge architectures and evaluation rubrics, and build machine learning models for automated quality assessment.
• Configure data collection workflows, investigate root causes of data-quality issues, research auditing methods, and put quality strategies and automated judging systems into practice.

Required qualifications

• Master’s degree in computer science, mathematics, statistics, machine learning, or an equivalent quantitative field.
• Programming experience in Java, C++, Python, or a related language; experience with SQL and a relational database management system, such as Oracle, or a data warehouse.

Preferred qualifications

• Experience implementing algorithms using both toolkits and self-developed code; publications in top-tier peer-reviewed conferences or journals.

The posting lists Bellevue, Washington, and Boston, Massachusetts, as locations. The stated Bellevue base salary range is USD 136,000.00–184,000.00 annually. The compensation package includes sign-on payments and restricted stock units; listed benefits include health coverage, 401(k) matching, paid time off, and parental leave.