PhysicsX is seeking a Senior Infrastructure Engineer to join their Research team in Singapore. The role focuses on building, scaling, and operating a high-performance computing (HPC) platform to support large-scale training and data generation for physics AI. The position is split between platform engineering and hands-on support for researchers. Responsibilities include designing and maintaining SLURM-on-Kubernetes infrastructure, managing GPU clusters, optimizing performance for distributed training, and developing automation tools. Required qualifications include 4-6 years of experience in infrastructure or HPC engineering, hands-on experience with SLURM and Kubernetes in production, and knowledge of high-performance networking (Infiniband, RDMA, NCCL). Candidates should be comfortable debugging complex stacks and working directly with researchers. Preferred qualifications include experience operating large-scale GPU clusters, background in CAE or simulation-driven data generation, and familiarity with infrastructure-as-code tools like Terraform and Helm. PhysicsX offers a hybrid work model and a collaborative, flat-structured environment.
For full details about this role, click “Apply on company site” to view the employer’s original posting.
A focused preparation guide, built around this job’s responsibilities and requirements.
✦ AI-generated guide
Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.
01
Understand the role
What matters most
SLURM-on-Kubernetes Platform Maintenance
The core of the role involves designing, building, and maintaining the specific orchestration stack used for large-scale distributed training. You must ensure the reliability and efficiency of this hybrid architecture, which is critical for supporting the Research team's CAE and simulation-driven data generation workflows.
Direct Research Support and Debugging
The role is split evenly between infrastructure engineering and hands-on support. You are expected to act as a partner to researchers, which requires the ability to diagnose complex failures across the stack—from scheduling and networking to individual job performance—and translate these technical findings into infrastructure improvements.
High-Performance Computing Optimization
You are responsible for maximizing the utilization and throughput of a cluster containing hundreds of GPUs. This requires deep knowledge of high-performance networking (Infiniband, RDMA, NCCL) and storage systems to ensure that data-intensive workloads do not become bottlenecks during model training.
Likely challenges to think through
Managing Hybrid Orchestration Complexity
Running SLURM on top of Kubernetes is a non-standard architecture that introduces unique operational overhead. Prepare by being ready to discuss how you handle the integration points between these two schedulers, specifically regarding resource isolation, containerization (Enroot/Pyxis), and fault tolerance for long-running jobs.
Balancing Engineering Roadmap with Reactive Support
Because the role is split between long-term platform building and immediate researcher support, you will face constant context switching. Prepare by demonstrating how you prioritize infrastructure automation and observability to reduce the time spent on manual debugging, allowing you to focus on scaling the platform.
03
Practice your answers
Technical practice
SLURM-on-Kubernetes Architecture
What to coverExplain how you would integrate SLURM with Kubernetes to manage HPC workloads. Discuss the trade-offs between running SLURM as a containerized service versus a native deployment and how you ensure resource isolation for GPU-intensive tasks.
Optimizing Distributed Training with NCCL
What to coverDescribe your approach to troubleshooting NCCL communication bottlenecks in a multi-node GPU cluster. Focus on how you verify Infiniband/RDMA connectivity and identify configuration issues that impact training throughput.
Containerization for HPC Workflows
What to coverCompare the use of Docker, Enroot, and Pyxis for HPC environments. Explain why specialized runtimes like Enroot are often preferred over standard Docker for high-performance research workloads.
GPU Cluster Health and Fault Tolerance
What to coverHow do you implement observability for long-running training jobs on hundreds of GPUs? Suggest strategies for detecting silent data corruption or hardware degradation before it ruins a multi-day training run.
Managing High-Performance Storage
What to coverDiscuss the challenges of supporting data-intensive CAE workloads on shared storage. Explain how you would balance storage performance requirements with the need for data reproducibility and accessibility across the cluster.
Infrastructure-as-Code for HPC
What to coverHow would you use Terraform or Helm to manage the lifecycle of an HPC cluster? Suggestion: Focus on how you maintain consistency across environments while allowing for the flexibility required by research teams.
Debugging Complex Stack Failures
What to coverWalk through your methodology for isolating a job failure that spans the application, container runtime, scheduler, and network layers. Emphasize the tools and logs you prioritize during the triage process.
Capacity Planning for Research Scaling
What to coverHow do you approach capacity planning when the research team's compute needs are unpredictable? Discuss how you balance current cluster utilization with the need to scale for future model training requirements.
Behavioural practice
Collaborating with Researchers
What to coverDescribe a time you had to translate a researcher's complex technical requirement into a concrete infrastructure improvement. Focus on how you managed expectations and ensured the solution actually improved their iteration speed.
Handling Critical Production Outages
What to coverTell me about a time you had to resolve a high-priority issue on a production cluster while under pressure. Explain your communication strategy with stakeholders and how you performed a post-mortem to prevent recurrence.
Influencing Technical Strategy
What to coverGive an example of a time you challenged an existing technical assumption or status quo within your team. Explain how you presented your case and the impact your suggestion had on the project or team efficiency.
Balancing Support and Engineering
What to coverThis role requires splitting time between hands-on support and long-term platform engineering. Describe how you prioritize these competing demands to ensure both immediate researcher needs and long-term infrastructure goals are met.
02
Sharpen your knowledge
Topics to prepare
SLURM-on-Kubernetes Architecture
Review the integration patterns for running SLURM schedulers atop Kubernetes clusters. Exercise: Design a conceptual diagram showing how a SLURM controller interacts with Kubernetes worker nodes to manage GPU resource allocation. This is critical as the role requires maintaining this specific hybrid orchestration stack.
Distributed Training Performance Optimization
Study the bottlenecks in large-scale model training, specifically regarding NCCL communication and RDMA/Infiniband throughput. Exercise: Analyze a hypothetical scenario where training throughput drops during multi-node synchronization and identify potential causes in the network or storage layer. This is relevant for the role's focus on debugging job failures and optimizing throughput.
Skills to revise
Containerization for HPC (Enroot/Pyxis)
Gain hands-on familiarity with Enroot and Pyxis for managing containerized research environments. Exercise: Create a simple Dockerfile and convert it to an Enroot image, then simulate a job submission using Pyxis to understand how it interfaces with SLURM. This is a core requirement for reproducible research workflows.
Infrastructure-as-Code (Terraform/Helm)
Practice defining cluster resources using Terraform and Helm charts. Exercise: Write a Helm chart to deploy a mock GPU-intensive service on a local Kubernetes cluster. This is highly relevant for the role's responsibility to build, scale, and automate infrastructure.
GPU Cluster Observability
Develop skills in monitoring GPU utilization and health. Exercise: Configure a basic Prometheus/Grafana dashboard to track GPU metrics like temperature, memory usage, and compute utilization. This is essential for the role's requirement to own cluster reliability and efficiency.
04
Your preparation checklist
Prepare a technical case study on SLURM-on-Kubernetes
Document a past project where you integrated SLURM with Kubernetes. Focus on how you managed resource scheduling, handled containerized workloads using Enroot or Pyxis, and ensured the stability of the orchestration layer for high-performance computing.
Draft a strategy for GPU cluster observability
Outline your approach to monitoring large-scale GPU clusters. Include specific metrics you track for NCCL performance, Infiniband throughput, and hardware health to proactively identify bottlenecks before they impact research training jobs.
Develop a framework for researcher support
Prepare examples of how you have translated complex infrastructure issues into actionable insights for non-infrastructure researchers. Describe your process for debugging job failures and optimizing throughput for distributed training workloads.
Formulate questions on HPC infrastructure scaling
Prepare thoughtful questions regarding the current infrastructure roadmap, such as how the team handles capacity planning for B200-scale clusters or how they balance the trade-offs between custom automation and off-the-shelf infrastructure-as-code tools like Terraform and Helm.
Availability last checked: 18 Oct 2026. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy
Share this opportunityKnow someone who would be a great fit?