Lead AI Platform Operations Engineer #AIDA

What matters most

• High Availability and Reliability of On-Prem AI Infrastructure

  The primary mandate is ensuring the stability of the OpenShift-based AI platform. You must demonstrate expertise in proactive monitoring, outage detection, and performance optimization to minimize downtime for critical AI workloads.

• Cybersecurity and Regulatory Compliance

  You are responsible for the security posture of the AIDA platform, requiring hands-on management of IAM, SIEM/SOAR, and vulnerability management. You must ensure all operations align with strict frameworks like ISO 27001, CIS, and NIST.

• Incident Management and Disaster Recovery

  You must lead root cause analysis and maintain robust disaster recovery strategies. This requires a strong SRE mindset to design systems that guarantee business continuity under pressure.

• Cross-Functional Collaboration and Automation

  You will bridge the gap between platform operations and MLOps/LLMOps teams. Success depends on your ability to integrate automation (Ansible) and observability (OpenTelemetry, Grafana) into AI/ML workflows.

Likely challenges to think through

• Balancing AI Workload Demands with On-Prem Constraints

  Managing GPU-intensive AI workloads on-premise often creates resource contention. Prepare to discuss how you optimize infrastructure for model hosting while maintaining platform stability and performance.

• Navigating High-Pressure Incident Response

  As a lead, you will be the point of escalation during critical outages. Prepare by reviewing your experience in leading teams through high-stakes troubleshooting and how you maintain clear communication during incident resolution.

• Integrating Security into Rapid AI Development Cycles

  There is a natural tension between the speed required by MLOps teams and the strict security/compliance requirements of a telco environment. Prepare to explain how you enforce security policies without becoming a bottleneck for AI innovation.

Topics to prepare

• Site Reliability Engineering (SRE) for AI Workloads

  Review SRE principles specifically applied to AI/ML infrastructure. Exercise: Draft a disaster recovery plan for an on-prem OpenShift cluster hosting large language models, focusing on RTO/RPO metrics and GPU resource failover. This is relevant because the role requires ensuring high availability for critical AI services.

• Security Compliance and Governance

  Study the implementation of ISO 27001, CIS benchmarks, and NIST frameworks within a containerized environment. Exercise: Create a checklist for conducting a security audit on an OpenShift cluster, identifying potential gaps in IAM and vulnerability management. This is relevant as the role involves overseeing security operations and regulatory alignment.

Skills to revise

• OpenShift Platform Operations

  Deepen knowledge of managing on-prem OpenShift clusters. Exercise: Simulate a node failure scenario and document the steps for automated recovery and performance monitoring using Grafana and OpenTelemetry. This is essential for the primary responsibility of maintaining platform reliability.

• Infrastructure-as-Code (Ansible)

  Practice writing Ansible playbooks for automated platform configuration and security hardening. Exercise: Develop a playbook to automate the deployment of security patches across multiple OpenShift nodes. This is relevant to the requirement for automation and operational excellence.

• AI/ML Infrastructure Management

  Understand the operational demands of GPU-accelerated workloads and model hosting. Exercise: Research and document the specific monitoring metrics required for GPU health and model inference latency. This is relevant to the role's focus on AI platform performance optimization.

• Map OpenShift and SRE experience to AI workloads

  Prepare a technical summary of your experience managing on-prem OpenShift clusters, specifically highlighting how you have implemented observability using Elastic Search, Grafana, and OpenTelemetry to maintain high availability for resource-intensive workloads like GPU-based AI models.

• Develop a disaster recovery and incident response case study

  Draft a structured walkthrough of a past critical incident you managed. Focus on your methodology for root cause analysis, your approach to designing disaster recovery strategies, and how you would apply these SRE practices to ensure business continuity for Singtel's AI platform.

• Align security operations with compliance frameworks

  Create a brief outline demonstrating your experience with ISO 27001, CIS, or NIST standards. Explain how you have integrated vulnerability management, IAM, and SIEM/SOAR tools into platform operations to secure sensitive data and infrastructure.

• Demonstrate automation proficiency with Ansible

  Prepare examples of Infrastructure-as-Code (IaC) scripts or Ansible playbooks you have authored to automate platform provisioning, security patching, or configuration management, emphasizing how these tools reduce manual toil in large-scale environments.

• Formulate strategic questions for the AIDA team

  Prepare thoughtful questions regarding the current challenges of scaling on-prem AI infrastructure, the specific integration points between the platform team and MLOps/LLMOps workflows, and how the team balances rapid innovation with strict regulatory compliance.