New opportunity

Senior Inference Runtime Engineer

Bitdeer Technologies Group · Singapore

About this role

Bitdeer Technologies Group is seeking a Senior Inference Runtime Engineer to own the performance-critical serving layer of their MaaS platform. The role focuses on optimizing the runtime stack for self-hosted LLMs to improve speed, cost-efficiency, and stability for OpenAI- and Anthropic-compatible APIs. Responsibilities include optimizing prefill/decode scheduling, continuous batching, KV cache behavior, speculative decoding, and long-context serving. You will tune runtimes like vLLM, SGLang, and TensorRT-LLM, profile bottlenecks across GPU memory, HBM bandwidth, and network paths, and lead model onboarding strategies including quantization and parallelism. You will also define runtime playbooks and collaborate with SRE and performance engineers to implement production improvements. Required qualifications include 6+ years of experience in systems, ML infrastructure, or high-performance backend engineering. Candidates must have hands-on experience with LLM serving runtimes, a strong understanding of GPU memory, CUDA/NCCL, and distributed inference, and proficiency in Go or Python. Experience operating production inference services with strict latency and availability targets is essential. The company offers an inclusive, fast-growing environment with opportunities for professional development and networking.

For full details about this role, click “Apply on company site” to view the employer’s original posting.

Skills for this role

LLM servingvLLMDynamoSGLangTensorRT-LLMTGITritonGPU memory optimizationCUDANCCLKV cacheDistributed inferenceGoPythonPerformance profilingML infrastructure

Checking your profile…

Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.

Understand the role

What matters most

Runtime Optimization and Tuning
The core responsibility is to enhance the performance of the MaaS platform by tuning runtimes like vLLM, SGLang, and TensorRT-LLM. You must demonstrate expertise in optimizing critical components such as continuous batching, KV cache management, and speculative decoding to meet strict latency and throughput targets.
Production-Grade Inference Operations
You are expected to own the stability and cost-efficiency of the serving layer. This requires translating low-level performance metrics into reliable production outcomes, including defining operational playbooks and managing model onboarding strategies like quantization and parallelism.
Deep Performance Profiling
You must be capable of identifying bottlenecks across the entire inference stack, including GPU memory utilization, HBM bandwidth limitations, and network/NCCL communication overheads, to ensure high-performance model serving.

Likely challenges to think through

Balancing Latency and Throughput
Optimizing for both low latency and high throughput often involves conflicting trade-offs in batching strategies and memory allocation. Prepare by discussing how you have navigated these trade-offs in previous production environments to maintain service level agreements.
Managing Distributed Inference Complexity
Scaling inference across multiple GPUs and nodes introduces significant complexity in communication and synchronization. Be prepared to discuss your experience with NCCL and distributed parallelism strategies, specifically how you mitigate performance degradation in distributed setups.
Rapid Model Onboarding and Stability
The need to onboard new models while maintaining existing service stability requires robust rollback strategies and safe default configurations. Prepare by outlining your methodology for validating new model deployments without impacting current production traffic.

Practice your answers

Technical practice

Optimizing KV Cache for Long-Context Serving

What to coverDiscuss the trade-offs between memory consumption and latency when managing KV cache for long-context LLMs. Explain how you would implement or tune techniques like PagedAttention or prompt caching to maintain high throughput without sacrificing stability.

Continuous Batching Strategies

What to coverExplain the mechanics of continuous batching in runtimes like vLLM or SGLang. Describe how you handle the scheduling of requests with varying sequence lengths to maximize GPU utilization and minimize idle time during the decode phase.

Profiling GPU Bottlenecks

What to coverDescribe your methodology for identifying performance bottlenecks in an inference stack. Specifically, how do you differentiate between HBM bandwidth limitations, compute-bound kernels, and NCCL communication overhead in a distributed inference setup?

Speculative Decoding Implementation

What to coverDiscuss the practical challenges of deploying speculative decoding in a production environment. Address how you balance the speedup gained from a draft model against the additional GPU memory and compute overhead required to run it.

Tensor and Pipeline Parallelism Trade-offs

What to coverWhen onboarding a new large-scale model, how do you decide between tensor parallelism and pipeline parallelism? Explain the impact of these strategies on latency, throughput, and the complexity of the underlying NCCL communication patterns.

Quantization for Production Inference

What to coverCompare different quantization methods (e.g., FP8, INT8, AWQ) in terms of their impact on model accuracy versus inference speed. Suggest a strategy for validating these models before rolling them out to production.

Debugging Distributed Inference Failures

What to coverIf a distributed inference service experiences intermittent latency spikes or NCCL timeouts, what steps do you take to isolate the root cause? Mention specific tools or metrics you would inspect across the network and GPU layers.

Runtime Selection and Customization

What to coverGiven the choice between vLLM, TensorRT-LLM, and SGLang, what criteria do you use to select the optimal runtime for a specific model architecture? Explain how you would approach modifying or extending these runtimes to meet unique platform requirements.

Behavioural practice

Translating Technical Metrics to Business Value

What to coverDescribe a situation where you had to justify a complex infrastructure optimization to non-technical stakeholders. How did you connect low-level performance improvements, such as latency reduction, to customer-visible reliability or cost-efficiency?

Collaborating with SRE for Production Stability

What to coverProvide an example of how you have partnered with SRE teams to improve the availability of a production service. Focus on how you defined runtime playbooks or safe defaults to prevent outages during high-traffic periods.

Managing Model Onboarding Risks

What to coverTell us about a time you led the onboarding of a new model into a production environment. How did you handle the transition, including testing, rollback strategies, and ensuring the model met strict latency targets?

Handling Technical Debt in High-Performance Systems

What to coverDescribe a time you identified a performance bottleneck that required significant refactoring of the existing runtime stack. How did you balance the need for immediate production stability with the long-term goal of improving system architecture?

Sharpen your knowledge

Topics to prepare

LLM Inference Optimization Techniques
Focus on the mechanics of continuous batching, KV cache management, and speculative decoding. Review how these techniques impact throughput and latency in high-demand production environments. This is directly relevant to the role's responsibility for optimizing the MaaS platform's performance-critical serving layer.
Distributed Inference and Parallelism
Study the trade-offs between tensor parallelism and pipeline parallelism when serving large models across multiple GPUs. Understand how NCCL facilitates communication in these setups. This is essential for the role's requirement to lead model onboarding and handle distributed inference.
GPU Memory and HBM Bandwidth Constraints
Deepen your understanding of how memory bandwidth limits LLM inference performance. Analyze how quantization and context length adjustments affect memory footprint and compute utilization. This knowledge is critical for profiling bottlenecks as requested in the job description.

Skills to revise

Runtime Profiling and Trace Analysis
Practice using profiling tools to identify bottlenecks in vLLM, SGLang, or TensorRT-LLM. Analyze execution traces to pinpoint latency spikes in prefill or decode phases. This directly supports the requirement to profile bottlenecks across GPU memory and network paths.
Production Runtime Tuning
Configure and benchmark an open-source LLM runtime (e.g., vLLM) with varying batch sizes and quantization levels. Document the impact on latency and throughput. This exercise prepares you for the role's focus on tuning runtimes for cost-efficiency and stability.
System-Level Programming in Python or Go
Review source code of common inference runtimes to understand their internal scheduling logic and API handling. Practice writing scripts to automate performance monitoring or model deployment tasks. This aligns with the requirement to be comfortable reading runtime source code and production metrics.

Your preparation checklist

Prepare a technical case study on LLM runtime optimization
Draft a summary of a past project where you improved inference performance. Focus on specific metrics like latency, throughput, or GPU utilization, and explain how you tuned parameters such as continuous batching, KV cache management, or quantization strategies.
Review source code and profiling traces for serving runtimes
Be prepared to discuss your experience reading source code for runtimes like vLLM, SGLang, or TensorRT-LLM. Practice explaining how you identified bottlenecks in GPU memory, HBM bandwidth, or NCCL communication during distributed inference.
Develop a framework for model onboarding playbooks
Create a sample checklist or 'playbook' you would use to onboard a new LLM into a production environment. Include considerations for runtime selection, tensor/pipeline parallelism, context length limits, and rollback strategies.
Formulate questions regarding production infrastructure
Prepare thoughtful questions for the team about their current MaaS platform architecture, such as how they handle multi-tenant isolation, their approach to monitoring streaming smoothness, or how they balance cost-efficiency with strict latency targets.

Availability last checked: 18 Oct 2026. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy

KEEP EXPLORING

Similar AI jobs

Related skills and specializations in Singapore. Matched to this role, not your profile.

Explore more jobs

Finding similar opportunities…