About this role
Amazon’s Artificial General Intelligence (AGI) team seeks an Applied Scientist to improve the quality of training and evaluation data for large language model and multimodal products. The role works closely with scientists developing Amazon Nova models, as well as engineers, domain experts and vendor teams.
Responsibilities
- Lead quality strategies and auditing frameworks for data collection workflows. Design standard operating procedures, quality metrics and sampling methods that help improve Nova’s benchmark performance.
- Conduct expert manual audits and meta-audits of auditor performance, provide targeted coaching, and investigate the root causes of data quality issues.
- Develop and maintain LLM-as-a-Judge systems, including judge architectures, evaluation rubrics and machine learning models for automated quality assessment.
- Configure data collection workflows, communicate quality feedback to stakeholders, research auditing methods and put quality strategies into practice.
Required qualifications
A master’s degree in computer science, mathematics, statistics, machine learning or an equivalent quantitative field; programming experience in Java, C++, Python or a related language; and experience with SQL and a relational database, such as Oracle, or a data warehouse.
Preferred qualifications
Experience implementing algorithms with both toolkits and self-developed code, and publications in top-tier peer-reviewed conferences or journals.
The posting lists Bellevue, Washington, and Boston, Massachusetts, as locations. The listed Bellevue base salary is USD 136,000–184,000 annually. The compensation package includes sign-on payments and restricted stock units; listed benefits include health coverage, 401(k) matching, paid time off and parental leave. Final compensation depends on experience, qualifications and location.