New opportunity

Senior System Software Engineer - LocalAI

NVIDIA · Pune, India

About this role

NVIDIA’s LocalAI team seeks a Senior System Software Engineer to develop efficient on-device AI software for RTX and DGX-class systems in Pune, India. The role focuses on low-latency local inference, efficient memory use, stable infrastructure and deployment on resource-constrained platforms.

Responsibilities

Work with software, research, architecture and product teams to align technical requirements and priorities for the AI ecosystem on RTX and DGX PCs.
Build and optimize the local inference stack for RTX, RTX Pro and DGX GPUs across different hardware architectures. Develop inference runtimes and execution stacks using tools such as llama.cpp, vLLM, PyTorch, WinML, DXCGC and TensorRT-RTX for LLM, vision-language, text-to-speech, automatic speech recognition and diffusion workloads.
Optimize models, data pipelines and runtimes for current and next-generation GPUs. Apply quantization, pruning, sparsity and distillation to deploy large models locally and on edge devices.
Debug systems, tune performance and assess performance–accuracy trade-offs. Build infrastructure for performance and accuracy sweeps, investigate gaps, drive fixes and establish guidelines for bringing new models and inference backends to production readiness.

Requirements

5+ years of experience and a bachelor’s, master’s or PhD in Computer Science, Software Engineering, Mathematics or a related field, or equivalent experience.
Strong C++ programming and debugging skills; foundations in data structures, algorithms and machine learning; and experience developing and optimizing inference pipelines and applications with ML/DL frameworks.
Deep knowledge of inference backends and runtime internals, including scheduling, memory management, KV-cache behavior, graph execution, quantization and hardware-aware optimization. Strong analytical, prioritization and written and verbal communication skills are required.

Preferred

Knowledge of modern machine learning, deep neural networks and generative AI, with relevant major open-source contributions; experience delivering end-to-end products across geographically distributed teams; low-level systems and GPU programming with CUDA; or contributions to inference runtimes, model tooling and performance infrastructure. Hands-on experience with Vulkan and DirectX is also valued.

The posting describes competitive salaries and a benefits package but gives no salary amount.

For full details about this role, click “Apply on company site” to view the employer’s original posting.

Skills for this role

C++Data structuresAlgorithmsMachine learningAI inferencellama.cppvLLMPyTorchWindows MLDXCGCTensorRTTensorRT-RTXGPU programmingCUDAVulkanDirectXQuantizationPruningSparsityKnowledge distillationKV-cache managementGraph executionPerformance tuningSystem-level debuggingCommunication

Checking your profile…

Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.

Understand the role

What matters most

Build the local inference stack
The role develops and optimizes runtimes and execution stacks for RTX, RTX Pro, and DGX GPUs across LLM, vision-language, speech, and diffusion workloads. Prepare to explain systems-level design decisions for on-device inference.
Optimize across the full pipeline
The posting calls for end-to-end optimization of models, data pipelines, and runtimes, backed by knowledge of scheduling, memory management, KV-cache behavior, and graph execution. Prepare examples that connect a measured bottleneck to a specific change and result.
Make large models practical locally
Low latency and efficient memory use on resource-constrained platforms are central goals. The duties explicitly include quantization, pruning, sparsity, and distillation; prepare to discuss how to evaluate their performance and accuracy effects.
Drive production readiness with evidence
The role builds performance and accuracy sweep infrastructure, investigates gaps, drives fixes, and sets guidelines for new models and backends. Prepare to describe a reproducible debugging and benchmarking approach.
Align technical priorities across teams
The engineer partners with software, research, architecture, and product teams to align requirements and priorities. Prepare to explain how technical findings can be communicated clearly enough to support a shared decision.

Likely challenges to think through

Balancing speed, memory, and accuracy
Likely preparation challenge, not a confirmed employer issue: a model optimization may reduce memory use or latency while degrading output quality. Prepare a concrete method for setting baselines, running accuracy and performance sweeps, and choosing among trade-offs.
Keeping behavior consistent across platforms
Likely preparation challenge: supporting multiple GPU architectures, runtimes, and workload types may make a fast path on one configuration unstable or ineffective on another. Prepare to discuss configuration-specific measurements, regression checks, and when to retain a fallback.
Locating end-to-end performance gaps
Likely preparation challenge: latency can arise in model execution, data movement, scheduling, or memory behavior, so a single benchmark may obscure the cause. Prepare a profiling and system-debugging sequence that isolates the bottleneck before proposing a fix.

Practice your answers

Technical practice

How would you find the main latency bottleneck in an on-device inference pipeline?

What to coverThe posting calls for end-to-end optimization of models, data pipelines and runtimes for low-latency local inference. Preparation suggestion: explain how you would measure each stage, distinguish compute from data movement and use the results to choose an optimization.

How would you design runtime scheduling for inference across different GPU architectures?

What to coverThe role calls for scheduling expertise and a stable, scalable inference stack across RTX, RTX Pro and DGX GPUs. Preparation suggestion: discuss how workload size, concurrency, latency and hardware capabilities affect scheduling choices and how you would validate them.

What KV-cache decisions matter when memory is constrained?

What to coverThe posting explicitly names KV-cache behavior, memory management and deployment on resource-constrained platforms. Preparation suggestion: address cache growth, allocation and reuse, and the trade-off between memory use and inference latency.

When could graph execution improve an inference runtime, and when might it be a poor fit?

What to coverGraph execution is listed among the required runtime internals. Preparation suggestion: discuss potential launch-overhead savings alongside costs such as setup, changing shapes or less flexible execution, and explain what you would measure.

How would you debug an intermittent C++ inference-runtime failure?

What to coverThe posting requires strong C++ debugging and system-level problem solving. Preparation suggestion: describe a method for reproducing the failure, narrowing it to a component, checking memory and concurrency behavior where relevant, and verifying a fix.

How would you compare quantization, pruning, sparsity and distillation for local deployment?

What to coverAll four techniques are named as ways to deploy large models locally or at the edge. Preparation suggestion: explain their different effects on memory, runtime speed and model quality, and why a measured gain on the target hardware matters.

How would you adapt an inference pipeline for LLM, vision-language, speech or diffusion workloads?

What to coverThe role supports LLM, vision-language, TTS, ASR and diffusion workloads. Preparation suggestion: pick two contrasting workloads and explain how their inputs, execution patterns and latency or memory needs would change your pipeline design.

What would guide your choice of inference framework or backend for a local AI application?

What to coverThe posting names llama.cpp, vLLM, PyTorch, Windows ML, DXCGC and TensorRT-RTX among relevant frameworks and execution stacks. Preparation suggestion: compare options you know using workload support, hardware fit, performance, integration effort and debugging needs rather than assuming one is universally best.

How would you build useful performance and accuracy sweeps for a new model or backend?

What to coverThe role includes building sweep infrastructure, investigating gaps and establishing production-readiness guidelines. Preparation suggestion: describe reproducible inputs, hardware and configuration coverage, comparable metrics, regression analysis and criteria for deciding what needs a fix.

When would lower-level GPU programming be worth pursuing instead of a runtime-level change?

What to coverHardware-aware optimization is required, while CUDA and other low-level GPU experience are listed as ways to stand out. Preparation suggestion: explain how profiling could justify a lower-level change and how you would weigh its measured benefit against portability and maintenance.

Behavioural practice

Tell me about a time you aligned teams with different technical priorities.

What to coverThe role involves working with software, research, architecture and product teams. Preparation suggestion: use a real situation, your actions to clarify requirements and resolve a trade-off, and the resulting decision or outcome.

Describe a time you had to prioritize several urgent engineering problems.

What to coverThe posting calls for managing multiple priorities in a fast-paced environment. Preparation suggestion: give a real situation, explain how you assessed impact and communicated your plan, and state the result.

Tell me about a difficult performance or stability issue you drove to resolution.

What to coverSystem-level debugging, gap analysis and driving fixes are stated duties. Preparation suggestion: describe the real symptoms, your investigative actions and how you verified the outcome without overstating your contribution.

Describe a time you communicated a technical trade-off to people with different backgrounds.

What to coverThe posting emphasizes written and verbal communication across engineering teams and management. Preparation suggestion: use a real example showing the situation, how you made the evidence and options clear, and what resulted.

Tell me about a time you helped bring a new system component into dependable use.

What to coverThe role includes guidelines for bringing new models and inference backends to production readiness. Preparation suggestion: describe a real situation, your actions to define checks or address gaps, and the observable result; do not imply experience with a specific NVIDIA process.

Sharpen your knowledge

Topics to prepare

Inference scheduling and KV-cache behavior
Revision exercise (suggestion): Explain how request scheduling and KV-cache allocation affect latency and memory use when several local LLM requests run concurrently. Posting relevance (explicit): The role requires deep understanding of scheduling, memory management and KV-cache behavior.
Graph execution and hardware-aware optimization
Revision exercise (suggestion): Trace an inference graph and identify where execution overhead or data movement could limit performance across GPU architectures. Posting relevance (explicit): The posting calls for graph-execution knowledge and optimization across RTX, RTX Pro and DGX GPUs.
Model optimization trade-offs
Revision exercise (suggestion): Compare when quantization, pruning, sparsity and distillation might reduce local deployment cost, and what accuracy checks each would need. Posting relevance (explicit): All four techniques and performance–accuracy analysis appear in the responsibilities.
Workload-specific inference requirements
Revision exercise (suggestion): Contrast latency, memory and execution considerations for an LLM and one of ASR, TTS, vision-language or diffusion inference. Posting relevance (explicit): The inference stack supports these workloads on local systems.
Production-readiness evidence
Revision exercise (suggestion): Define what performance and accuracy sweep results would reveal a regression in a new model or backend, and how to prioritize investigation. Posting relevance (explicit): The role develops sweeps, investigates gaps and establishes bring-up guidelines.

Skills to revise

C++ systems debugging
Revision exercise (suggestion): Debug a small C++ inference-stage example with a memory-lifetime or concurrency fault; explain the root cause and fix. Posting relevance (explicit): Excellent C++ programming and debugging and system-level debugging are required.
Inference pipeline development
Revision exercise (suggestion): Walk through or implement a minimal local inference pipeline using a framework named in the posting, such as llama.cpp or PyTorch, and identify its preprocessing, execution and output stages. Posting relevance (explicit): Developing and optimizing inference pipelines with ML/DL frameworks is required.
Performance and accuracy measurement
Revision exercise (suggestion): Design a repeatable sweep that records latency, memory use and an accuracy measure across model configurations; explain how you would isolate a regression. Posting relevance (explicit): The role builds sweep infrastructure and uses results to drive fixes.
Applying model optimizations
Revision exercise (suggestion): Apply quantization to a small inference example, compare its memory and latency with the baseline, and check output quality. Posting relevance (explicit): Quantization and performance–accuracy trade-off analysis are part of the role.
Communicating engineering priorities
Revision exercise (suggestion): Write a brief recommendation that summarizes an inference bottleneck, supporting measurements, a proposed fix and its trade-offs for software and product stakeholders. Posting relevance (explicit): The role aligns priorities across teams and requires strong written and verbal communication.

Your preparation checklist

Prepare an inference pipeline walkthrough
The role calls for experience developing and optimizing inference applications. Choose a project you can discuss and prepare a diagram covering the model, framework, execution path, GPU constraints, changes made and measured results.
Rehearse a C++ debugging case
The posting requires strong C++ and debugging skills. Prepare a concrete account of a difficult failure or performance regression: how you isolated it, the fix and the evidence that it worked.
Sketch a local inference runtime design
As interview practice, sketch how you would serve an LLM on an RTX-class system with low latency and limited memory. Explain scheduling, KV-cache management, graph execution and what you would test across hardware architectures.
Build a performance–accuracy comparison
The work includes performance and accuracy sweeps. Prepare a small table or proposed test plan showing latency, memory use and accuracy before and after an optimization such as quantization; state measurement conditions and trade-offs.
Explain production-readiness decisions
Prepare a brief example or hypothetical plan for investigating a backend performance gap, prioritizing fixes and setting checks for stability and new-model bring-up. Distinguish work you have done from your proposed approach.
Prepare collaboration examples and employer questions
The role spans software, research, architecture and product teams. Outline one example you can substantiate of aligning competing priorities, then ask which workloads and backends matter most and how the team evaluates production readiness.

Availability last checked: 5 Nov 2026. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy

KEEP EXPLORING

Similar AI jobs

Related skills and specializations in India. Matched to this role, not your profile.

Explore more jobs

Finding similar opportunities…