New opportunity

Applied Scientist, Agent Evaluation & Adaptive Model Routing

Bitdeer Technologies Group · Singapore

About this role

The Applied Scientist for Agent Evaluation & Adaptive Model Routing will join the Bitdeer AI Lab to build evaluation and decision systems that make agentic inference measurable, reliable, and adaptive. This role is central to the lab's mission of turning energy into affordable intelligence by optimizing the economics of model serving. Responsibilities include owning and extending LLM and agent evaluation pipelines, developing representative task suites, and measuring performance at the trajectory level across metrics such as task success, tool use, quality, cost, latency, and token consumption. You will research and prototype adaptive model-routing strategies, including rule-based baselines, cascades, stage-aware routing, uncertainty-aware selection, and escalation and recovery policies. The role involves taking methods from offline evaluation through traffic replay, shadow testing, and internal pilots, while collaborating with MaaS and platform engineering teams on production integration. Qualifications include a Bachelor’s, Master’s, or PhD in Computer Science, Machine Learning, Statistics, Electrical Engineering, or a related field, with substantial hands-on experience in LLM evaluation, agentic systems, or adaptive inference. Candidates must possess strong Python programming skills, practical experience with PyTorch, and a track record of building reliable experimental pipelines. Demonstrated experience in designing evaluation pipelines for LLMs or agents, including automated scoring and failure analysis, is required. Implementation-level depth in at least one core area such as model selection, uncertainty calibration, or cascading is expected. Preferred qualifications include experience with multi-model APIs, agent harnesses, production model monitoring, and familiarity with inference systems like vLLM or SGLang. Publications at top-tier ML/NLP venues or significant open-source contributions are highly valued. The role requires strong ownership and the ability to drive research questions from problem definition to 0-to-1 prototypes.

For full details about this role, click “Apply on company site” to view the employer’s original posting.

Skills for this role

PythonPyTorchLLM EvaluationAgentic SystemsApplied Machine LearningAdaptive InferenceModel RoutingUncertainty EstimationPreference ModelingContextual BanditsOnline LearningvLLMSGLangA/B TestingStatistical Analysis

Checking your profile…

Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.

Understand the role

What matters most

Development of Agentic Evaluation Pipelines
The role requires building and extending comprehensive evaluation frameworks that measure agent performance at the trajectory level. This includes defining metrics for task success, tool-use accuracy, latency, and token consumption to ensure agentic inference is both measurable and reliable.
Adaptive Model-Routing Strategy Research
You must research and prototype routing strategies such as cascades, uncertainty-aware selection, and escalation policies. The priority is to optimize the economics of model serving by dynamically selecting the most cost-effective and performant model for specific tasks.
Bridging Research to Production
The candidate must own the lifecycle of routing methods from offline evaluation through traffic replay, shadow testing, and internal pilots. Success depends on collaborating with platform engineering teams to integrate these prototypes into production environments.

Likely challenges to think through

Balancing Cost-Quality Pareto Frontiers
A core difficulty is moving beyond aggregate model scores to optimize for the trade-off between inference cost and task quality. Prepare by demonstrating how you have used statistical analysis to justify routing decisions that maintain performance while minimizing resource consumption.
Managing Ambiguity in 0-to-1 Prototyping
The role demands taking loosely defined research questions and turning them into validated, working prototypes. Prepare by highlighting specific instances where you defined the scope of an experimental project and established clear success criteria in an unstructured environment.
Complexity of Agentic Failure Analysis
Evaluating multi-turn, tool-using agents involves diagnosing complex planning failures and recovery behaviors. Prepare by discussing your methodology for automated scoring and root-cause analysis in agentic systems, specifically how you isolate failures in tool calls versus reasoning.

Practice your answers

Technical practice

Designing Trajectory-Level Evaluation Pipelines

What to coverExplain how you would design an evaluation framework for multi-turn, tool-using agents. Focus on how to measure task success, tool-call correctness, and recovery behavior, and suggest how to handle failure analysis when an agent deviates from the optimal path.

Constructing Cost-Quality Pareto Frontiers

What to coverDescribe your approach to balancing model performance against cost and latency. Discuss how you would visualize and analyze these trade-offs to make informed decisions about model routing rather than relying on aggregate accuracy scores.

Adaptive Model Routing Strategies

What to coverCompare and contrast different routing strategies such as cascades, stage-aware routing, and uncertainty-aware selection. Suggest how you would implement a baseline and evaluate its effectiveness against more complex, adaptive policies.

Uncertainty Calibration in LLMs

What to coverDiscuss methods for estimating and calibrating uncertainty in LLM outputs. Explain how this uncertainty information can be utilized to trigger escalation or recovery policies within an agentic system.

Production Integration and Shadow Testing

What to coverDetail the process of moving an offline evaluation prototype into a production environment. Explain the role of traffic replay and shadow testing in validating your routing policies before full deployment.

Optimizing Inference with vLLM or SGLang

What to coverExplain how you would leverage features like prompt caching or specific reasoning controls in systems like vLLM or SGLang to improve agent performance. Suggest how these system-level optimizations impact your evaluation metrics.

Statistical Rigor in Experimental Design

What to coverHow do you ensure your evaluation results are statistically significant and reproducible? Suggest how to structure controlled comparisons and define honest baselines to avoid common pitfalls in LLM benchmarking.

Contextual Bandits for Online Learning

What to coverDescribe how you would apply contextual bandits or online learning techniques to improve model routing over time. Discuss the challenges of exploration versus exploitation in a live agentic inference environment.

Behavioural practice

Driving 0-to-1 Research Prototypes

What to coverDescribe a situation where you identified an ambiguous research problem and drove it to a working prototype. Focus on how you defined the problem, validated your results, and gained internal buy-in.

Collaborating Across Engineering Teams

What to coverProvide an example of working with platform or infrastructure engineers to integrate a research-heavy feature into a production system. How did you manage competing priorities and technical constraints?

Handling Evaluation Failures

What to coverTell me about a time when your evaluation pipeline revealed unexpected model behavior or failures. How did you analyze the root cause and what actions did you take to improve the system's reliability?

Managing Trade-offs in Ambiguous Environments

What to coverDescribe a project where you had to make a difficult decision between model quality, latency, and cost. How did you justify your choice to stakeholders and what was the final outcome?

Sharpen your knowledge

Topics to prepare

Adaptive Model Routing Strategies
Focus on the conceptual frameworks for model selection, including cascading, stage-aware routing, and uncertainty-aware selection. Exercise: Design a theoretical routing policy that balances cost and latency for a multi-model agent system, specifically defining the criteria for escalating a task from a smaller, cheaper model to a more capable one.
Agentic Evaluation Methodologies
Understand how to measure performance at the trajectory level rather than just output quality. Exercise: Define a set of metrics for a multi-turn tool-using agent, focusing on tool-call correctness, planning failure rates, and recovery behavior, and explain how you would automate the scoring of these metrics.
Cost-Quality Pareto Frontiers
Learn to analyze the trade-offs between inference cost, latency, and model performance. Exercise: Create a mock analysis comparing three different model configurations across a test suite, plotting them on a Pareto frontier to identify the optimal model for specific task complexity levels.

Skills to revise

Python and PyTorch Experimental Pipelines
Develop robust, reproducible code for ML experiments. Exercise: Build a modular Python script that runs a batch of prompts through multiple model APIs, logs the latency and token consumption, and saves the results into a structured format for subsequent failure analysis.
Inference System Familiarity (vLLM/SGLang)
Gain practical knowledge of high-throughput inference engines. Exercise: Review the documentation for vLLM or SGLang and draft a configuration setup that optimizes for prompt caching and tool-calling efficiency, explaining how these settings impact overall system throughput.
Statistical Analysis for A/B Testing
Apply rigorous statistical methods to validate model improvements. Exercise: Design a shadow testing plan for a new routing policy, detailing how you would use statistical significance testing to compare the performance of the new policy against a rule-based baseline.

Your preparation checklist

Develop a Cost-Quality Pareto Frontier Analysis
Prepare a case study or project walkthrough demonstrating how you have balanced LLM performance metrics (quality, task success) against operational constraints (latency, token consumption, cost). Show how you visualize these trade-offs to inform model selection or routing decisions.
Design an Agentic Evaluation Framework
Outline a methodology for evaluating multi-turn, tool-using agents. Focus on how you measure trajectory-level success, identify planning failures, and automate scoring for tool-call correctness. Be prepared to discuss how you handle failure analysis and regression testing in complex agentic workflows.
Prepare a Technical Deep-Dive on Adaptive Routing
Select one core area of expertise—such as uncertainty calibration, cascading, or stage-aware routing—and prepare a brief explanation of how you would implement it. Explain the logic behind your chosen baseline and how you would validate the strategy using offline evaluation or shadow testing.
Formulate Strategic Questions for the Lab
Prepare questions regarding the current state of their agent harness, the specific production challenges they face with model serving economics, and how they prioritize research-to-production transitions. This demonstrates your focus on practical, high-impact engineering.

Availability last checked: 18 Oct 2026. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy

KEEP EXPLORING

Similar AI jobs

Related skills and specializations in Singapore. Matched to this role, not your profile.

Explore more jobs

Finding similar opportunities…