About this role
Distyl AI seeks an Applied AI Researcher on its Benchmarking team to define how progress in enterprise AI systems is measured. The team develops evaluations grounded in real-world operational complexity, informing decisions about new architectures, techniques and releases.
Responsibilities
- Design evaluation frameworks that measure reasoning depth, interaction quality, reliability and operational impact; build benchmarks that capture real-world complexity.
- Research adversarial robustness testing, longitudinal performance tracking and human-in-the-loop assessment. Investigate how metrics influence model behavior and develop rigorous ways to quantify emerging capabilities.
- Produce findings that guide Distyl’s research priorities and contribute to broader evaluation standards.
Qualifications
- Experience building or maintaining benchmarks, test suites or experimental frameworks for model or system performance.
- Ability to design fair, reproducible experiments and analyze noisy results, supported by strong programming, data analysis and prototyping skills.
- A demonstrated record of research results and experience building intelligent systems with models rather than primarily training or fine-tuning them. The posting highlights expertise in compound AI systems, agentic collaboration and techniques such as ensembling, ReAct and graph-of-thoughts.
- Daily use of AI tools such as ChatGPT, Cursor and Perplexity to accelerate work, and an ability to demonstrate practical results to enterprise customers. Researchers may come from varied academic backgrounds; no specific degree or number of years of experience is stated.
Location and compensation: The role is listed in San Francisco and New York and follows a hybrid schedule of at least three in-office days per week (Tuesday–Thursday). Base salary is $150K–$250K, depending on experience, location and level, with equity and benefits including fully covered medical, dental and vision insurance for employees and dependents, flexible time off, retirement and financial planning resources, wellness and family-building benefits, and in-office lunches and snacks.