Senior AI Data Engineer #AIDA

What matters most

• Operationalizing RAG and Agentic AI

  The role requires building and maintaining knowledge base stacks for GenAI. You must demonstrate expertise in vectorization, lifecycle management of knowledge bases, and the ability to scale these solutions for enterprise-wide agentic AI use cases.

• Production-Grade Pipeline Engineering

  You are expected to own the end-to-end lifecycle of batch and streaming pipelines using Databricks and Kafka. This includes ensuring high performance, reliability, observability, and maintaining comprehensive operational runbooks for incident management.

• Enterprise Governance and Security Compliance

  All data solutions must adhere to strict enterprise data architecture, cloud security, and access control policies. You must integrate these governance requirements directly into the CI/CD workflows and design phase to ensure compliance.

• Stakeholder Technical Leadership

  Beyond technical execution, you must lead discussions with architects, product owners, and source-system teams. This involves translating complex business requirements into maintainable platform patterns and providing technical guidance to other engineers.

Likely challenges to think through

• Balancing Innovation with Enterprise Rigor

  Implementing cutting-edge GenAI and RAG solutions often conflicts with rigid enterprise security and governance policies. Prepare by demonstrating how you have successfully navigated compliance requirements while deploying experimental or advanced AI technologies in a production environment.

• Managing Complex Hybrid Cloud Data Ecosystems

  The role involves integrating diverse sources (APIs, databases, streaming) across a hybrid cloud platform using tools like Microsoft Fabric and Delta Lake. Prepare by highlighting your experience in troubleshooting cross-platform data quality issues and maintaining metadata traceability in large-scale environments.

• Scaling Agentic AI Reliability

  Agentic AI introduces non-deterministic behaviors that are difficult to monitor. The challenge lies in building observability frameworks that can detect failures in RAG retrieval or agent logic. Prepare by discussing how you design monitoring systems that go beyond simple pipeline health to ensure the quality of AI-generated outputs.

Topics to prepare

• Retrieval-Augmented Generation (RAG) Architecture

  Review the end-to-end lifecycle of RAG systems, specifically focusing on embedding generation, vector database storage, and retrieval strategies. Exercise: Design a conceptual architecture diagram for a RAG pipeline that handles document ingestion, vectorization, and retrieval for an agentic AI use case, ensuring you account for data security and access control policies.

• Enterprise Data Governance and Security

  Understand how to apply security, access control, and compliance policies within a hybrid cloud data platform. Exercise: Draft a checklist for ensuring a new data pipeline meets enterprise compliance standards, focusing on data masking, encryption at rest/transit, and audit logging requirements.

• Production-Grade Pipeline Operations

  Focus on observability, incident management, and the creation of operational runbooks. Exercise: Create a template for an incident response runbook for a failed streaming pipeline, including steps for root-cause analysis, data reconciliation, and stakeholder communication.

Skills to revise

• PySpark and Databricks Optimization

  Master performance tuning for large-scale data processing. Exercise: Optimize a sample PySpark job by identifying potential data skews, implementing efficient partitioning strategies, and utilizing Delta Lake features to improve query performance.

• Kafka Streaming Integration

  Develop proficiency in building and maintaining streaming data pipelines. Exercise: Configure a Kafka producer and consumer pattern that handles schema evolution and ensures exactly-once processing semantics for a real-time data ingestion task.

• CI/CD for Data Pipelines

  Implement automated deployment workflows for data and AI components. Exercise: Define a CI/CD pipeline configuration that includes automated unit testing for PySpark transformations, linting, and deployment to a staging environment using version control.

• Prepare a RAG architecture walkthrough

  Develop a technical summary of a RAG solution you have implemented, focusing on vectorization, knowledge base storage, and retrieval optimization. Be prepared to discuss how you handled data lifecycle management and embedding strategies for agentic AI use cases.

• Review production-grade pipeline patterns

  Prepare to discuss your experience with Databricks and Kafka in a production environment. Focus on how you have implemented observability, error handling, and performance tuning for batch and streaming pipelines to ensure high reliability.

• Draft examples of CI/CD and governance integration

  Prepare concrete examples of how you have integrated CI/CD workflows into data engineering projects. Highlight your experience applying enterprise security, access control, and data governance policies to ensure compliance in cloud-scale analytics.

• Formulate questions for the AIDA team

  Prepare thoughtful questions regarding the current data architecture stack at Singtel, the specific challenges of scaling RAG solutions for their business domains, and how the AIDA unit balances operational stability with the need for rapid AI innovation.

• Refine technical communication examples

  Prepare to describe a scenario where you led a technical discussion with non-technical stakeholders or source-system owners to define ingestion patterns. Emphasize your ability to translate complex data requirements into maintainable platform solutions.