Sr. Inference Service SRE & Automation Engineer

What matters most

• Achieving 99.95% SLA for Inference Endpoints

  The core mandate is to ensure high availability for the Infermesh platform. This requires rigorous SLI/SLO instrumentation, error budget management, and proactive capacity planning to handle GPU saturation, KV cache occupancy, and request queuing without service degradation.

• Automating Incident Remediation and AIOps

  Success depends on transitioning from manual intervention to autonomous workflows. You must build a robust remediation actuator and workflow engine, ensuring that every human-led incident response is codified into self-healing platform logic.

• Resilience and Failure Mode Mitigation

  You must own the end-to-end reliability of the inference path. This involves creating a comprehensive resilience matrix, conducting regular fault injection and failover drills, and ensuring automated detection and mitigation for all critical failure modes across the GPU fleet and Kubernetes infrastructure.

• GPU Fleet and Inference Runtime Operations

  Maintaining the health of high-performance GPU hardware (H200/B300/GB300) is critical. You are responsible for managing inference runtimes like vLLM or TensorRT-LLM, handling model rollouts, and diagnosing complex hardware-level issues such as NVLink or NCCL failures.

Likely challenges to think through

• Managing Complex GPU Hardware Failures

  GPU fleet operations involve nuanced hardware signals (DCGM, Xid, ECC). Preparing for this requires demonstrating experience in isolating hardware faults without impacting customer traffic, specifically through automated node draining and replica recovery strategies.

• Balancing Reliability with Rapid Deployment

  Implementing strict change safety protocols, such as canary rollouts and automated rollbacks for models and configurations, is difficult in a high-throughput environment. Prepare by discussing how you balance the need for speed with the risk of regression in a production-critical API.

• Cross-Cluster Traffic Engineering

  Managing multi-cluster serving across different regions (MY, SG, JP) introduces significant latency and synchronization challenges. You should be prepared to discuss strategies for cross-cluster traffic steering, DNS/GSLB failover, and maintaining consistency across distributed Kubernetes environments.

Topics to prepare

• SLA and Error Budget Management

  Focus on the conceptual framework of defining SLIs/SLOs for high-availability inference services. Exercise: Draft a strategy for a 99.95% SLA, identifying which metrics (TTFT, TPOT, error rates) constitute a breach and how to manage error budgets to balance reliability work with feature development.

• Resilience and Failure Mode Analysis

  Understand how to build a failure-mode and resilience matrix for complex distributed systems. Exercise: Create a hypothetical matrix for an LLM inference path, mapping critical failure modes (e.g., GPU hang, KV cache exhaustion) to detection signals, mitigation strategies, and RTO targets.

• AIOps and Automated Remediation

  Explore the shift from manual runbooks to autonomous workflows. Exercise: Design a logic flow for an automated remediation controller that detects a failing GPU replica via DCGM signals and triggers a safe drain and replacement process without human intervention.

Skills to revise

• LLM Inference Runtime Operations

  Gain hands-on familiarity with vLLM, SGLang, or TensorRT-LLM. Exercise: Configure a local inference engine to monitor KV cache occupancy and batching behavior, then simulate a concurrency limit breach to observe how the system handles load shedding.

• GPU Fleet Health Monitoring

  Develop proficiency in diagnosing hardware-level issues in high-performance computing environments. Exercise: Practice interpreting DCGM, Xid, and ECC error logs to distinguish between transient GPU faults and permanent hardware failures requiring node evacuation.

• Kubernetes Traffic Engineering

  Master advanced networking and routing for multi-cluster environments. Exercise: Use Helm and GitOps (ArgoCD) to deploy a service mesh or ingress configuration that demonstrates cross-cluster failover capabilities for a latency-sensitive API.

• Map LLM Inference Reliability Challenges

  Prepare a technical summary of how you have managed inference runtimes like vLLM or TensorRT-LLM in production. Focus on your experience with KV cache management, batching strategies, and mitigating latency issues like TTFT (Time to First Token) and TPOT (Time Per Output Token) to meet strict SLAs.

• Design a GPU Fleet Failure Recovery Workflow

  Draft a high-level architecture for automated GPU health monitoring using DCGM. Explain how you would implement an automated drain-and-cordon process for nodes experiencing ECC errors or NCCL communication failures without impacting customer inference requests.

• Develop an SLI/SLO Framework for API Services

  Create a sample SLI/SLO dashboard design for a customer-facing API. Include how you would define error budgets, set burn-rate alerts, and use synthetic probing to validate the end-to-end reliability of the inference path.

• Prepare a Multi-Cluster Traffic Engineering Strategy

  Outline a strategy for cross-cluster traffic steering and failover. Describe how you would use tools like Kubernetes, service meshes, or GSLB to ensure high availability across different geographic regions (MY/SG/JP) while maintaining 99.95% uptime.

• Formulate Targeted Questions for the Interviewers

  Prepare questions regarding their current AIOps maturity, such as: 'What are the most frequent manual interventions currently required for Infermesh?' or 'How does the team currently balance feature development with the 99.95% SLA reliability mandate?'