About this role
Run human labeling and evaluation of AI outputs, maintain benchmark datasets and golden sets, categorize failure modes, and partner with QA, engineering, and product teams to improve model and product quality.
Skills for this role
Data labelingData analysisQuality assuranceEvaluating AI-generated outputsNatural Language Processing (NLP)SearchRecommendation systemsMachine LearningQualitative analysisSQLData retrievalSpreadsheet workflowsWritten communicationVerbal communicationCross-functional collaborationLabeling guidelinesRubricsDataset maintenanceBenchmarkingHuman evaluationCalibrationLLM evaluationFailure mode analysisHallucination detectionRetrieval failure analysisTool use evaluation