About this role
Build and scale systems to ingest, process, and deliver large-scale video and multimodal datasets for model training, owning the full pipeline from raw content to training-ready datasets. Partner with curation and modeling teams to operationalize dataset recipes, implement containerized infrastructure, and optimize cloud storage, orchestration, and curation pipelines for improved model outcomes and cost-efficiency.
Skills for this role
Dataset curationLarge-scale data pipelinesDistributed data processingPySparkRayWorkflow orchestrationAirflowJob schedulingMonitoring and failure recoveryKubernetesDockerAWSGCSAzureCloud storage optimizationData versioningCachingData retentionImage-text datasetsVLM-based captioningMetadata generationQuality scoring modelsAesthetic scoring modelsCLIPEmbedding-based filteringSemantic filteringDeduplicationFFmpegPyAVDALI (NVIDIA) or equivalent media tools (DALI) - shortened to DALI? Nokeep 'DA