New opportunity

Senior Software Engineer

Rakuten Asia Pte. Ltd. · Singapore

About this role

As an LLM Inference Optimization Engineer within the AI & Data Division (AIDD), you will focus on maximizing the performance, efficiency, and scalability of LLM inference workloads on Rakuten's GPU clusters. You will deeply optimize inference engines and their underlying GPU kernels to ensure AI models are served at peak efficiency, reducing latency and maximizing throughput. Key Responsibilities: Profile, optimize, and extend modern LLM inference engines (e.g., vLLM, SGLang, TensorRT-LLM). Design and implement inference-time optimizations including quantization, KV-cache management, continuous batching, speculative decoding, and various parallelism strategies. Develop and tune GPU kernels (CUDA, Triton) for critical operators like attention, GEMM, and MoE. Build benchmarking and load-testing frameworks to measure performance against SLOs. Partner with infrastructure teams to deploy and scale inference on Kubernetes-based GPU clusters. Mandatory Qualifications: 3+ years of hands-on experience optimizing deep learning inference on NVIDIA GPUs, preferably for LLMs. Strong knowledge of LLM inference internals (attention, KV cache, batching, quantization). Proficiency in CUDA and/or Triton kernel programming. Strong software engineering fundamentals in Python and C++. Bachelor's or higher degree in Computer Science, Engineering, or a related field. Nice-to-Have Skills: Experience with open-source ML systems, advanced inference techniques (MoE, disaggregated prefill/decode, long-context optimization), low-precision inference (FP8, INT4, FP4), GPU performance profiling tools (Nsight, PyTorch Profiler), and deep learning training frameworks (PyTorch, DeepSpeed, Megatron-LM, FSDP).

For full details about this role, click “Apply on company site” to view the employer’s original posting.

Skills for this role

LLM inferencevLLMSGLangTensorRT-LLMTriton Inference ServerCUDATritonPythonC++GPU kernel programmingQuantizationKV-cache managementContinuous batchingSpeculative decodingKubernetesPyTorchDeepSpeedMegatron-LMFSDPNsight SystemsPyTorch Profiler

Checking your profile…

Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.

Understand the role

What matters most

LLM Inference Engine Optimization
The core responsibility is to profile and extend engines like vLLM, SGLang, or TensorRT-LLM. You must demonstrate the ability to improve throughput and latency by implementing advanced techniques such as continuous batching, KV-cache management, and speculative decoding.
Custom GPU Kernel Development
You are expected to write and tune high-performance kernels using CUDA or Triton. This requires a deep understanding of modern NVIDIA GPU architectures to optimize critical operators like attention mechanisms, GEMM, and MoE layers.
Scalable Infrastructure Integration
You must bridge the gap between model optimization and production deployment by partnering with infrastructure teams to manage inference on Kubernetes-based GPU clusters, ensuring reliability, autoscaling, and adherence to SLOs.

Likely challenges to think through

Balancing Performance vs. Generalization
Optimizing kernels for specific operators often risks creating brittle code that fails when model architectures change. Prepare by discussing how you maintain modularity in your CUDA/Triton code while achieving peak hardware utilization.
Managing Inference SLOs in Dynamic Environments
Maintaining strict latency and throughput targets in a shared Kubernetes cluster is difficult due to resource contention. Prepare by explaining your approach to benchmarking, load testing, and capacity planning under realistic, fluctuating traffic patterns.
Keeping Pace with Rapid Research Cycles
The LLM inference landscape evolves weekly. You will face the challenge of deciding which new research techniques (e.g., new quantization methods or parallelism strategies) are stable enough for production. Prepare by describing your framework for evaluating and adopting new technologies.

Practice your answers

Technical practice

Optimizing KV Cache Management

What to coverExplain the trade-offs between different KV cache eviction strategies and memory management techniques in high-throughput LLM serving. Discuss how you would minimize memory fragmentation while maintaining low latency during continuous batching.

CUDA and Triton Kernel Development

What to coverDescribe your process for profiling and optimizing a custom GPU kernel, such as a fused attention operator. Mention how you utilize tools like Nsight Systems or PyTorch Profiler to identify bottlenecks like memory bandwidth or compute utilization.

Inference Engine Architecture

What to coverCompare the architectural approaches of vLLM, SGLang, and TensorRT-LLM regarding request scheduling and batching. Explain when you would choose one over the others for a specific production workload.

Quantization and Low-Precision Inference

What to coverDiscuss the impact of using FP8, INT4, or FP4 precision on model accuracy versus inference throughput. Explain how you would implement and validate these quantization techniques in a production environment.

Parallelism Strategies for LLMs

What to coverExplain the differences between tensor parallelism, pipeline parallelism, and expert parallelism in the context of MoE models. Describe how you decide which strategy to apply based on model size and hardware constraints.

Speculative Decoding Implementation

What to coverExplain the mechanism of speculative decoding and how it improves inference latency. Address the challenges of selecting an appropriate draft model and managing the acceptance rate in a production serving pipeline.

Benchmarking and SLOs

What to coverDescribe how you design a load-testing framework to measure end-to-end performance against strict SLOs. Suggest metrics beyond simple latency and throughput that are critical for evaluating production inference stability.

Kubernetes GPU Cluster Deployment

What to coverDiscuss the challenges of deploying large-scale inference services on Kubernetes, specifically regarding GPU resource scheduling, autoscaling, and fault tolerance. Suggest how to handle node failures without violating latency SLOs.

Behavioural practice

Handling Technical Trade-offs

What to coverDescribe a situation where you had to balance model performance against infrastructure costs or latency requirements. Explain your decision-making process and how you communicated the trade-offs to stakeholders.

Staying Current with Research

What to coverThe LLM inference field moves rapidly; provide an example of a new research technique you recently implemented or evaluated. Explain how you assessed its viability for production use.

Cross-Team Collaboration

What to coverDescribe a time you partnered with an infrastructure or DevOps team to resolve a production bottleneck. Focus on how you aligned on technical requirements and ensured a successful deployment.

Problem-Solving Under Pressure

What to coverShare an example of a critical performance regression you encountered in a production inference system. Detail the steps you took to diagnose the root cause and the actions you implemented to prevent recurrence.

Sharpen your knowledge

Topics to prepare

LLM Inference Internals
Focus on the mechanics of attention mechanisms, KV cache management, and continuous batching. Review how these components impact latency and throughput in large-scale deployments. Exercise: Diagram the data flow of a request through a transformer model, identifying where memory bottlenecks occur during the prefill and decode phases.
Parallelism Strategies for LLMs
Understand the implementation of tensor, pipeline, and expert parallelism for MoE models. This is critical for scaling inference across multiple GPUs. Exercise: Compare the trade-offs between tensor parallelism and pipeline parallelism regarding communication overhead and memory distribution.
Low-Precision Inference
Study quantization techniques including FP8, INT4, and FP4. Understand how these methods reduce memory footprint and increase throughput without significant accuracy loss. Exercise: Research the impact of different quantization schemes on GPU kernel execution efficiency.

Skills to revise

CUDA and Triton Kernel Programming
Develop proficiency in writing custom kernels for operators like GEMM and attention. This is a mandatory requirement for optimizing inference engines. Exercise: Write a simple Triton kernel to perform a matrix operation and profile it using Nsight to identify potential memory access optimizations.
Inference Engine Profiling
Gain hands-on experience with Nsight Systems, Nsight Compute, or PyTorch Profiler to identify performance bottlenecks in inference workloads. Exercise: Profile a standard vLLM or TensorRT-LLM deployment to identify the primary source of latency in a specific model serving pipeline.
Kubernetes for GPU Clusters
Understand the deployment and scaling of inference services on Kubernetes, focusing on GPU resource management and observability. Exercise: Create a configuration manifest for deploying an inference service on a Kubernetes cluster that includes resource requests and limits for NVIDIA GPUs.

Your preparation checklist

Document GPU Optimization Experience
Prepare a technical summary of your past work optimizing LLM inference on NVIDIA GPUs. Focus on specific metrics improved, such as latency reduction or throughput increases, and detail your experience with vLLM, SGLang, or TensorRT-LLM.
Review Kernel Programming Proficiency
Be prepared to discuss your experience writing custom CUDA or Triton kernels. Have examples ready of how you optimized critical operators like attention or GEMM, and explain your understanding of modern GPU architecture constraints.
Prepare a System Design Walkthrough
Outline a scenario where you deployed or scaled an inference service on Kubernetes. Include how you handled challenges like continuous batching, KV-cache management, or SLO-driven capacity planning.
Formulate Technical Questions for the Team
Prepare questions regarding Rakuten's current inference stack, such as the specific GPU clusters used, the primary LLM architectures currently being served, and how the team balances research-driven optimizations with production stability.

Availability last checked: 18 Oct 2026. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy

KEEP EXPLORING

Similar AI jobs

Related skills and specializations in Singapore. Matched to this role, not your profile.

Explore more jobs

Finding similar opportunities…