About this role
Own the evaluation strategy, process, and platform to make Rakuten's AI product quality measurable, prevent regressions, guide release decisions, and continuously improve AI experiences through reliable data and insights. Define evaluation criteria, prioritize platform capabilities (dataset management, model/prompt comparison, automated and human grading, multi-turn evaluation, trace review, dashboards, release integration), and connect evaluations with production monitoring, A/B testing, feedback, and business KPIs.
Skills for this role
Product ManagementAI/ML Product ManagementEvaluation StrategyEvaluation Criteria DefinitionDataset ManagementModel VersioningPrompt VersioningModel ComparisonPrompt ComparisonAutomated GradingHuman GradingHuman EvaluationMulti-turn EvaluationTrace ReviewDashboard DesignRelease IntegrationData PipelinesA/B TestingProduction MonitoringUser Feedback AnalysisLarge-scale Data AnalysisPlatform RoadmappingGovernanceCross-functional CollaborationCommunication Skills