Senior Interactive Vision Model Researcher

What matters most

• Architecting Foundation Models for Video Generation

  The core of this role involves designing and iterating on next-generation video foundation architectures. You must demonstrate deep expertise in both diffusion models and autoregressive generation, specifically focusing on how to scale these architectures effectively for high-quality video output.

• Optimizing Long-Context and Tokenization Efficiency

  A critical priority is solving the computational bottlenecks of long-video generation. This requires advanced knowledge of KV Cache compression, long-context attention mechanisms, and the development of high-compression-ratio video tokenizers to handle multi-resolution and multi-frame-rate data.

• Leading Large-Scale Pre-training and Scaling Research

  You will be responsible for defining data mixture strategies, curriculum learning, and exploring Scaling Laws. This requires a proven ability to lead end-to-end training projects from scratch using large-scale distributed systems in PyTorch.

Likely challenges to think through

• Balancing Research Innovation with Product R&D Constraints

  As this role sits within a game development studio, you may face the challenge of aligning cutting-edge academic research with the practical, performance-oriented requirements of game engines. Prepare by discussing how you have previously adapted theoretical model architectures to meet specific product-level constraints or latency requirements.

• Managing Distributed Training Complexity

  Training foundation models from scratch involves significant infrastructure challenges. You should be prepared to discuss your experience in debugging distributed training, handling hardware-level bottlenecks, and optimizing resource allocation for massive token-scale pre-training.

Topics to prepare

• Scaling Laws for Generative Models

  Review the empirical relationship between compute, dataset size, and model parameters in video generation. Exercise: Draft a research plan outlining how you would measure performance saturation points when scaling a video foundation model to ensure efficient resource allocation.

• Long-Context Attention and KV Cache Optimization

  Study mechanisms for handling long-sequence video data, specifically focusing on memory-efficient attention and KV cache compression. Exercise: Compare the trade-offs between different attention variants (e.g., FlashAttention, Ring Attention) in the context of maintaining temporal consistency in long-video generation.

• Data Mixture and Curriculum Learning Strategies

  Understand how to curate and sequence training data to improve model convergence and generalization. Exercise: Design a curriculum learning strategy for a video foundation model that transitions from low-resolution, short-clip pre-training to high-resolution, long-form generation.

Skills to revise

• Large-Scale Distributed Training in PyTorch

  Demonstrate proficiency in training models across multiple nodes and GPUs. Exercise: Optimize a PyTorch training script to minimize communication overhead during distributed data-parallel training, specifically addressing bottlenecks in gradient synchronization.

• Video VAE and Tokenizer Design

  Master the trade-offs between compression ratio, reconstruction quality, and temporal coherence in video tokenizers. Exercise: Analyze the impact of different latent space dimensions and quantization methods on the final visual quality of generated video frames.

• Diffusion and Autoregressive Model Implementation

  Implement core architectures for both diffusion-based and autoregressive video generation. Exercise: Build a minimal prototype in PyTorch that demonstrates a specific sampling technique or architectural modification relevant to high-fidelity video synthesis.

• Synthesize your research portfolio

  Prepare a summary of your first-author publications in video generation or diffusion models. Be ready to discuss the specific architectural choices, training data strategies, and the impact of your work on the field.

• Prepare a technical deep-dive on Video VAEs and Tokenizers

  Develop a presentation or discussion points detailing your experience with high-compression-ratio video tokenizers. Focus on the trade-offs between reconstruction quality, compression ratio, and computational efficiency.

• Review large-scale distributed training methodologies

  Document your experience with PyTorch-based distributed training at scale. Prepare to explain how you have handled challenges related to data parallelism, model parallelism, and training stability in large-scale foundation model projects.

• Formulate a perspective on long-video generation

  Draft a technical analysis of current long-context attention mechanisms and KV Cache compression techniques. Be prepared to discuss how these technologies can be applied to improve video generation consistency and length.

• Prepare questions on the Lightspeed Tech Center roadmap

  Draft thoughtful questions regarding the team's current approach to Scaling Laws, their data mixture strategies for pre-training, and how they balance academic research with the practical requirements of game development.