Research Scientist / Research Engineer, Inference Systems

What matters most

• Bridging Research and Production

  The role requires translating frontier research in inference systems into production-grade code. You must demonstrate the ability to take theoretical improvements in areas like tiered caching or MoE models and implement them as stable, high-performance solutions within the existing serving stack.

• Rigorous Performance Measurement

  You must be capable of defining meaningful metrics before optimization and critically evaluating performance data. The role demands an honest assessment of what specific benchmarks prove or fail to prove, ensuring that system-level changes provide tangible benefits for real-world fleet workloads.

• System-Level Implementation

  Proficiency in Python and C++ is essential for building and modifying inference engines. You will be expected to contribute directly to the serving stack, requiring deep technical knowledge of frameworks like vLLM or SGLang to design new approaches where current ones are insufficient.

Likely challenges to think through

• Adapting to Agentic Workload Patterns

  Agentic workloads create unpredictable traffic patterns that differ significantly from standard LLM requests. Prepare by researching how disaggregation and dynamic resource allocation can mitigate the latency and throughput bottlenecks inherent in these complex, multi-step agentic interactions.

• Balancing Research Ambition with Production Stability

  The expectation to both publish research and ship to production creates a tension between experimental innovation and operational reliability. Prepare by discussing your methodology for iterative development, such as how you use staging environments or canary deployments to validate research ideas without disrupting the AI Cloud platform.

Topics to prepare

• LLM Inference Serving Architectures

  Study modern serving paradigms including disaggregation, tiered caching, and Mixture-of-Experts (MoE) model execution. Exercise: Compare the architectural trade-offs of centralized versus disaggregated serving stacks for high-throughput agentic workloads.

• Performance Metrics for Inference Systems

  Understand how to define and validate system-level performance metrics beyond simple latency, such as throughput, token generation efficiency, and resource utilization. Exercise: Design a benchmarking suite for a hypothetical MoE model to identify bottlenecks in memory bandwidth versus compute.

Skills to revise

• Inference Engine Implementation (vLLM/SGLang)

  Gain hands-on experience with the internals of vLLM or SGLang to understand how they manage KV cache and request scheduling. Exercise: Modify a custom scheduling policy or add a diagnostic hook in an open-source inference engine to measure cache hit rates.

• Systems Programming in C++ and Python

  Demonstrate proficiency in writing high-performance code that bridges Python research prototypes with C++ production backends. Exercise: Optimize a Python-based inference component by porting a performance-critical section to C++ using pybind11 or similar interfaces.

• Distributed Serving Systems

  Develop expertise in scaling inference across multiple nodes and GPUs. Exercise: Simulate a distributed inference scenario to analyze the impact of network latency on inter-node communication during tensor parallelism.

• Prepare a technical portfolio of inference system contributions

  Compile a summary of your hands-on experience with vLLM, SGLang, or similar frameworks. Highlight specific system-level improvements you implemented, focusing on how you addressed challenges like tiered caching, MoE model serving, or disaggregated architectures.

• Develop a rigorous performance measurement methodology

  Be prepared to discuss how you define success metrics for inference workloads. Create a case study demonstrating how you established a baseline, identified a bottleneck, implemented an optimization, and validated the results with honest, data-driven analysis.

• Review recent literature on agentic and distributed inference

  Stay current on the latest research regarding agentic workload traffic patterns and distributed serving. Prepare to discuss which recent academic or industry papers you find most impactful and how their findings could be applied to production-scale inference stacks.

• Refine your C++ and Python systems implementation examples

  Select two code samples that demonstrate your proficiency in both languages, specifically within the context of high-performance computing or systems programming. Ensure you can explain the trade-offs made in your design choices and how you managed memory or concurrency.

• Formulate strategic questions for the AI Lab team

  Prepare questions regarding the current challenges the AI Cloud platform team faces with production workloads, the specific research priorities of the lab, and how the team balances the dual requirements of shipping production code and publishing research findings.