New opportunity

Data Science Manager (CXD)

Rakuten Asia Pte. Ltd. · Singapore

About this role

The Data Science Manager will lead the applied AI science function within Rakuten's CXD team, managing a team of data scientists and ML engineers responsible for the intelligence layer of the company's conversational AI platform. This role is based in Singapore and involves close collaboration with product and engineering teams across Japan and the APAC region to drive measurable improvements in model quality, retrieval performance, and personalisation. The position is chartered around two foundational responsibilities: the Runtime Decision Plane (mathematical and scientific sign-off) and the Learning Plane (feedback systems). Key Responsibilities: Team Leadership & Science Strategy: Build, manage, and mentor a team of data scientists and ML engineers; define technical roadmaps and establish rigorous scientific standards for the group-wide Customer Service AI platform. LLM Strategy & Model Lifecycle Management: Own the end-to-end lifecycle of production LLMs, including vendor evaluation, selection, fine-tuning, and version upgrades; shift focus from prompt optimisation to systematic model comparison and agentic reasoning evaluation. RAG Architecture & Continuous Iteration: Lead research and improvement of RAG systems, including retrieval quality, reranking, and knowledge base health; drive structured iteration cycles based on offline and online evaluation. Evaluation Framework, Experimentation & VoC Intelligence: Define multi-layer evaluation frameworks, including A/B experiments and production monitoring; establish pre-launch evaluation gates and manage Voice of Customer analysis to create feedback loops. Conversational Memory & Personalisation: Set technical direction for session state and user profile modelling; partner with stakeholders to deliver AI-driven personalisation for upsell and conversion. Mandatory Requirements: 7+ years in applied machine learning or NLP, including 3+ years in a people management or technical lead role. PhD in Computer Science, Machine Learning, NLP, or related discipline is strongly preferred. Deep hands-on experience with LLMs and RAG systems in production, including retrieval pipeline design and reranking. Demonstrated ability to provide mathematical and algorithmic sign-off on system design. Proficiency in Python and cloud ML infrastructure (GCP preferred). Strong communication skills for cross-regional stakeholder management.

For full details about this role, click “Apply on company site” to view the employer’s original posting.

Skills for this role

Machine LearningNLPLLMsRAGPythonGCPVector SearchEmbedding ModelsA/B TestingBanditsInterleavingAgentic ReasoningPeople Management

Checking your profile…

Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.

Understand the role

What matters most

Establishing Scientific Rigor and Governance
The role requires providing formal mathematical and algorithmic sign-off on system designs, moving beyond heuristic-based tuning to defensible, theory-grounded production decisions. You must establish and enforce pre-launch evaluation gates, such as hallucination rate ceilings and intent accuracy floors, to ensure platform integrity across multiple business units.
Scaling the Learning Plane and Feedback Loops
You are responsible for building a systematic 'Learning Plane' that functions as a continuous feedback flywheel. This involves creating error taxonomies, active learning pipelines, and robust evaluation frameworks that translate production telemetry and Voice of Customer signals into measurable, iterative improvements for the conversational AI platform.
Cross-Regional Stakeholder Management
Success depends on effectively collaborating with product and engineering teams across Singapore and Japan. You must be able to synthesize complex analytical findings into clear product problem statements and communicate the science function's value to both technical and non-technical senior stakeholders across different cultural and regional contexts.
Optimizing LLM and RAG Production Lifecycle
You must lead the end-to-end lifecycle of production LLMs and RAG systems, including vendor selection, fine-tuning, and complex retrieval pipeline design. The priority is to shift the team's focus from simple prompt optimization to systematic model comparison, agentic reasoning evaluation, and balancing cost-quality-latency trade-offs.

Likely challenges to think through

Balancing Rapid Iteration with Scientific Standards
A core tension exists between the need for rapid deployment of new business unit features and the requirement for rigorous, pre-launch scientific validation. Prepare to discuss how you maintain a high 'scientific bar' without becoming a bottleneck for product delivery, specifically by automating evaluation gates and establishing clear, repeatable 'ways of working'.
Attributing Root Causes in Complex RAG Systems
Distinguishing between retrieval failures and LLM synthesis failures is a stated requirement. Prepare to explain your methodology for isolating these components within a multi-step orchestration environment, as misattributing these errors can lead to ineffective optimization cycles.
Aligning Personalization with Business Outcomes
The role requires shifting the chatbot from a support tool to a growth channel focused on upsell and conversion. The challenge is to build a scientific framework that connects behavioral signals and LTV (Lifetime Value) to specific AI-driven recommendations, ensuring that personalization efforts are directly tied to measurable revenue impact rather than just answer quality.

Practice your answers

Technical practice

RAG System Evaluation and Root Cause Analysis

What to coverExplain how you distinguish between retrieval failures and LLM synthesis failures in a RAG pipeline. Discuss specific metrics or diagnostic techniques you use to isolate whether the issue lies in the retrieval quality, reranking, or the generation model's reasoning.

Agentic Reasoning Evaluation Frameworks

What to coverHow do you move beyond simple prompt optimization to evaluate multi-step agentic reasoning? Describe the methodologies you use to benchmark orchestration strategies and chain-of-thought accuracy in production environments.

Mathematical Sign-off on System Design

What to coverDescribe your process for providing formal mathematical and algorithmic sign-off on complex ML system designs. How do you justify routing policies or retrieval scoring mechanisms to stakeholders while balancing cost, quality, and latency?

Online Experimentation for Conversational AI

What to coverDiscuss the trade-offs between A/B testing, interleaving, and multi-armed bandit approaches when optimizing conversational AI features. Provide an example of how you would design an experiment to measure the impact of a new personalization model on conversion rates.

Managing the LLM Lifecycle

What to coverOutline your strategy for managing the end-to-end lifecycle of production LLMs, including vendor selection, fine-tuning, and version upgrades. How do you ensure stability and performance consistency when transitioning between model versions?

Error Taxonomies and Feedback Loops

What to coverExplain how you build and maintain error taxonomies to inform active learning pipelines. How do you ensure that production telemetry and Voice of Customer signals are systematically integrated into the model improvement cycle?

Conversational Memory and Context Modeling

What to coverWhat are the technical challenges in maintaining session state and long-term user profiles for large-scale conversational AI? Discuss how you handle contextualized retrieval while ensuring compliance with data privacy requirements.

Personalization and LTV-Driven Modeling

What to coverHow do you shift a conversational AI from a support tool to a growth channel using personalization? Describe how you incorporate behavioral signals and lifetime value (LTV) metrics into the model's decision-making framework.

Behavioural practice

Cross-Regional Stakeholder Management

What to coverDescribe a situation where you had to align technical strategy with stakeholders across different regions or cultures. How did you communicate complex scientific concepts to non-technical leaders to gain their buy-in and influence project prioritization?

Mentoring and Team Capability Building

What to coverGive an example of how you have mentored data scientists or ML engineers to improve their technical rigor. How do you balance the need for rapid iteration with the requirement to maintain high scientific standards within your team?

Handling Conflicting Priorities

What to coverTell me about a time when you had to manage a conflict between business expansion commitments and the need for rigorous scientific evaluation. How did you decide on the trade-offs and ensure the platform remained stable?

Influencing Product Strategy

What to coverProvide an example of how you synthesized analytical findings into a clear product problem statement that successfully influenced leadership. What was the outcome of your recommendation on the product roadmap?

Sharpen your knowledge

Topics to prepare

RAG System Evaluation and Iteration
The role requires distinguishing retrieval failures from synthesis failures. Exercise: Design a diagnostic framework that separates retrieval metrics (e.g., Recall@K, MRR) from generation metrics (e.g., faithfulness, answer relevance) to identify root causes in a multi-step RAG pipeline. This is critical for the 'Learning Plane' responsibility.
Agentic Reasoning and Orchestration
The position demands moving beyond prompt engineering to formal evaluation of agentic workflows. Exercise: Define a set of quantitative benchmarks for multi-step agent routing policies and chain-of-thought reasoning. This addresses the requirement for scientific sign-off on orchestration strategies.
Online Experimentation Frameworks
The role involves managing A/B tests, interleaving, and bandit algorithms. Exercise: Draft a strategy for conducting online experiments in a conversational AI context, specifically focusing on how to measure the impact of personalisation on conversion and LTV rather than just chatbot accuracy.

Skills to revise

Production LLM Lifecycle Management
The role requires owning the end-to-end lifecycle including vendor selection and fine-tuning. Exercise: Create a decision matrix for evaluating LLM vendors based on cost, latency, and quality trade-offs for a customer service use case, incorporating specific 'pre-launch gates' for deployment.
Error Taxonomy and Feedback Loops
The role requires building systems that transform production telemetry into learning cycles. Exercise: Develop a structured error taxonomy for a conversational AI platform that categorizes failures into intent misclassification, retrieval gaps, or hallucination, and map these to specific remediation actions.
Mathematical Sign-off on System Design
The role requires justifying architectural decisions formally. Exercise: Prepare a technical brief that defends a specific reranking or retrieval scoring mechanism using mathematical justification, simulating the 'scientific sign-off' process required for cross-regional stakeholders.

Your preparation checklist

Prepare a RAG system design case study
Develop a technical walkthrough of a production-grade RAG system you have led. Focus on your methodology for retrieval quality, reranking strategies, and how you distinguished between retrieval failures and LLM synthesis errors during evaluation.
Draft a framework for LLM evaluation and governance
Create a document outlining how you would implement pre-launch evaluation gates for a conversational AI platform. Include your approach to defining metrics for hallucination rates, intent accuracy, and cost-quality-latency trade-offs.
Articulate your scientific leadership philosophy
Prepare examples of how you have provided mathematical and algorithmic sign-off on complex system designs. Be ready to discuss how you balance rigorous scientific standards with the need for rapid iteration in a cross-regional team environment.
Develop a strategy for feedback-driven learning loops
Outline a plan for building an 'error taxonomy' and active learning pipeline. Explain how you would use Voice of Customer (VoC) data to systematically improve model performance and inform the product roadmap.
Formulate questions for cross-regional stakeholders
Prepare thoughtful questions regarding the current challenges of the conversational AI platform, the specific technical hurdles in scaling across diverse business units, and how the team currently manages collaboration between Singapore and Japan.

Availability last checked: 18 Oct 2026. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy

KEEP EXPLORING

Similar AI jobs

Related skills and specializations in Singapore. Matched to this role, not your profile.

Explore more jobs

Finding similar opportunities…