New opportunity

Agent Reliability & Evaluation Algorithm Engineer

Shopee · Singapore

About this role

The Engineering and Technology team at Shopee is seeking an Agent Reliability & Evaluation Algorithm Engineer to design and own the evaluation system for LLM Agents. This role involves building benchmark construction, implementing LLM-as-Judge frameworks with human annotation, and managing regression/comparison pipelines to guide model optimization. Responsibilities include: Responsibilities: - Design and own the evaluation system for LLM Agents, including benchmark construction, LLM-as-Judge with human annotation, and regression/comparison pipelines. - Continuously monitor product metrics to guide Agent and model optimization. - Build Agent reliability analysis, including error attribution and uncertainty estimation across multi-step trajectories, and turn failure cases into targeted improvements. - Drive evaluation-guided Agent optimization, including Harness design (planning, verification, self-refine), tool calling, context engineering, and memory. - Lead evaluation-driven post-training of LLMs for vertical domains (SFT, DPO, RL), including data pipelines and reward model design. - Build and optimize RAG/retrieval systems and the Agent tool layer (search, code interpreters, sandboxes, MCP) and evaluate their impact on end-to-end Agent quality. - Explore frontier research in Agent reliability and evaluation and drive a full closed loop from research to deployment. Requirements: - Master’s degree or above in Artificial Intelligence, Computer Science, Mathematics, or a related field. - At least 1 year of full-time industry experience building and deploying production LLM Agent systems (task planning, orchestration, tool calling), including designing evaluation sets and metrics. - Hands-on experience fine-tuning LLMs via SFT and DPO (GRPO/RL is a plus), and building/optimizing RAG/retrieval systems. - Good programming skills; proficient in Python (SQL and PyTorch are a plus). - Familiarity with training and inference frameworks such as verl, vLLM, FSDP, or LLaMA-Factory. - Strong analytical skills, with a solid grasp of statistics and experiment design.

For full details about this role, click “Apply on company site” to view the employer’s original posting.

Skills for this role

LLM AgentsLLM-as-JudgeBenchmark constructionSFTDPORLRAGPythonSQLPyTorchverlvLLMFSDPLLaMA-FactoryTask planningOrchestrationTool callingError attributionUncertainty estimation

Checking your profile…

Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.

Understand the role

What matters most

Design and Ownership of LLM Agent Evaluation Systems
The core priority is building a robust evaluation framework that includes benchmark construction, LLM-as-Judge pipelines, and human-in-the-loop annotation. You must demonstrate the ability to create metrics that accurately reflect agent performance across multi-step trajectories and use these to drive model optimization.
Evaluation-Driven Agent Optimization and Post-Training
You are expected to bridge the gap between evaluation results and model improvement. This involves leading post-training efforts (SFT, DPO, RL) and optimizing agent components like planning, tool calling, and memory based on specific failure cases identified through your reliability analysis.
End-to-End Reliability and Error Attribution
A critical requirement is the ability to perform deep-dive reliability analysis. You must be able to attribute errors within complex agent workflows, estimate uncertainty, and translate these findings into actionable improvements for the RAG systems and tool-calling layers.

Likely challenges to think through

Balancing Automated Evaluation with Human Annotation
Designing an 'LLM-as-Judge' framework that remains reliable at scale is difficult. Prepare by discussing how you manage the trade-off between the speed of automated evaluation and the high cost/accuracy of human annotation, specifically how you handle bias and calibration in judge models.
Attributing Errors in Multi-Step Agent Trajectories
In complex agent systems, it is often unclear whether a failure stems from the model's reasoning, the tool-calling layer, or the retrieved context. Prepare to explain your methodology for isolating these variables and how you use uncertainty estimation to decide when an agent should retry or escalate a task.
Closing the Loop from Research to Production
The role requires driving a full closed loop from frontier research to deployment. The challenge lies in maintaining high-quality, stable production performance while simultaneously experimenting with new techniques like GRPO or advanced RAG architectures. Be ready to discuss how you prioritize stability in a fast-moving, hyper-growth environment.

Practice your answers

Technical practice

Designing LLM-as-Judge Frameworks

What to coverExplain how you would design an LLM-as-Judge system to evaluate multi-step agent trajectories. Discuss strategies for mitigating bias in the judge model, incorporating human-in-the-loop annotation, and ensuring the judge's alignment with ground-truth metrics.

Error Attribution in Multi-step Agent Trajectories

What to coverDescribe your approach to identifying the root cause of failure in an agentic workflow. How do you distinguish between errors originating from task planning, tool calling, or context retrieval, and how do you quantify uncertainty in these steps?

Optimizing RAG Systems for Agentic Workflows

What to coverDiscuss techniques for optimizing retrieval performance in an agentic context, such as embedding fine-tuning or re-ranking strategies. Explain how you evaluate the impact of retrieval quality on the end-to-end reliability of the agent.

Post-training Strategies: SFT and DPO

What to coverCompare SFT and DPO for improving agent performance in vertical domains. Detail your process for constructing high-quality data pipelines and designing reward models to guide the optimization of agent behavior.

Agent Harness Design and Self-Refinement

What to coverHow do you implement self-refinement and verification mechanisms within an agent harness? Explain the trade-offs between planning-heavy architectures and reactive, tool-calling-focused designs.

Scaling Inference and Training Frameworks

What to coverDescribe your experience with frameworks like vLLM, FSDP, or LLaMA-Factory. How do you manage memory and throughput constraints when fine-tuning or deploying large-scale agent models?

Benchmark Construction for Agent Evaluation

What to coverExplain your methodology for building a robust benchmark suite for LLM agents. How do you ensure the benchmark covers diverse edge cases and provides actionable insights for model iteration?

Statistical Rigor in Experiment Design

What to coverHow do you apply statistical methods to validate improvements in agent performance? Discuss how you handle variance in non-deterministic agent outputs when comparing different model versions or system configurations.

Behavioural practice

Driving Research to Deployment

What to coverDescribe a situation where you transitioned an agentic research concept into a production environment. What were the primary technical hurdles, and how did you collaborate with cross-functional teams to ensure a successful deployment?

Handling Failure in Production Systems

What to coverProvide an example of a significant failure in an LLM agent system you managed. How did you perform the post-mortem, attribute the error, and implement a systemic fix to prevent recurrence?

Balancing Innovation and Reliability

What to coverTell us about a time you had to balance the need for exploring frontier AI research with the requirement for stable, reliable production performance. How did you decide which experimental features were ready for the product?

Collaborative Problem Solving

What to coverDescribe a scenario where you had to align different stakeholders on the definition of 'agent quality' or evaluation metrics. How did you reconcile conflicting viewpoints to drive a unified optimization strategy?

Sharpen your knowledge

Topics to prepare

LLM Agent Evaluation Frameworks
Focus on designing robust evaluation systems for multi-step agent trajectories. Exercise: Draft a proposal for an LLM-as-Judge pipeline that incorporates human-in-the-loop annotation and regression testing to measure agent reliability. This is critical for the role's requirement to own the evaluation system and drive model optimization.
Agent Reliability and Error Attribution
Study methods for analyzing failure cases in autonomous agents, specifically uncertainty estimation and error attribution. Exercise: Create a case study on how to diagnose a failure in a tool-calling agent, identifying whether the error originated from the planning, retrieval, or execution phase.
Evaluation-Driven Post-Training
Understand the lifecycle of using evaluation metrics to guide SFT, DPO, and RL processes. Exercise: Outline a data pipeline that converts agent failure logs into high-quality training samples for DPO or SFT to improve specific vertical domain performance.

Skills to revise

LLM Fine-Tuning and Alignment
Gain hands-on experience with SFT and DPO using frameworks like LLaMA-Factory or FSDP. Exercise: Fine-tune a small model on a custom dataset using LLaMA-Factory and evaluate the impact of DPO on model alignment, directly supporting the requirement for evaluation-driven post-training.
RAG and Tool Layer Optimization
Develop proficiency in building and evaluating RAG systems and tool-calling layers (e.g., search, code interpreters). Exercise: Implement a simple agent that utilizes a tool-calling framework to solve a multi-step task, then measure the impact of different retrieval strategies on the agent's success rate.
Python for AI Engineering
Ensure proficiency in Python for building production-grade agent systems. Exercise: Write a script to automate the evaluation of an agent's tool-calling accuracy using a library like vLLM for efficient inference, reflecting the need for strong programming skills in a production environment.

Your preparation checklist

Prepare a case study on LLM-as-Judge implementation
Document a past project where you designed an evaluation framework for LLM agents. Focus on how you constructed benchmarks, implemented LLM-as-Judge, and integrated human-in-the-loop annotation to validate model performance.
Demonstrate technical proficiency in Agent optimization
Be prepared to discuss your experience with Agent Harness design, specifically regarding task planning, self-refine mechanisms, and tool-calling reliability. Explain how you used error attribution and uncertainty estimation to diagnose and fix multi-step trajectory failures.
Showcase post-training and RAG expertise
Prepare to explain your workflow for fine-tuning LLMs using SFT and DPO, including how you designed reward models. Additionally, detail your experience optimizing RAG systems, specifically regarding retrieval quality, ranking, and embedding fine-tuning.
Review framework and infrastructure familiarity
Refresh your knowledge of training and inference frameworks such as vLLM, FSDP, or LLaMA-Factory. Be ready to explain how you have utilized these tools in production environments to deploy scalable agent systems.
Formulate strategic questions for the team
Prepare questions regarding the current challenges in Shopee's agent reliability, such as how they balance automated evaluation with human feedback at scale, or how they prioritize tool-layer improvements to impact end-to-end agent quality.

Availability last checked: 18 Oct 2026. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy

KEEP EXPLORING

Similar AI jobs

Related skills and specializations in Singapore. Matched to this role, not your profile.

Explore more jobs

Finding similar opportunities…