New opportunity

Principal Data Engineer (AWS, Databricks, Ai, ML Flow, Data Architecture, Apache Airflow)

Mastercard · Pune, India

About this role

Mastercard Foundry R&D is seeking a Principal Data Engineer to design and build the data foundations powering advanced analytics, machine learning, generative AI, and agentic AI solutions. In this role, you will collaborate with AI Engineers, Data Scientists, Researchers, Software Engineers, and Product Managers to create reusable data capabilities that accelerate experimentation and innovative product delivery.

Responsibilities

Lead the architecture, design, and engineering of scalable batch, streaming, and event-driven data platforms.
Design and build reliable data ingestion, transformation, enrichment, processing, and consumption pipelines.
Develop governed lakehouse and modern data platform architectures using cloud-native technologies and distributed data processing frameworks.
Build and operate platforms that support analytics, machine learning, generative AI, and agentic AI workloads.
Create reusable data products, services, APIs, and self-service capabilities that accelerate experimentation and production delivery.
Enable the full AI and machine learning lifecycle through robust feature engineering, training, evaluation, deployment, and inference pipelines.
Optimize data workloads for performance, scalability, reliability, resiliency, and cost efficiency.
Define engineering standards for data modeling, software development, testing, CI/CD, observability, and operational excellence.
Implement data governance capabilities including quality controls, lineage, metadata management, cataloging, access control, retention, and compliance.
Apply security and privacy-by-design principles for sensitive and regulated data environments.
Evaluate emerging data and AI technologies through prototypes, proof-of-concepts, and technical assessments.
Translate loosely defined research, innovation, or product requirements into practical architectures and incremental delivery plans.
Provide hands-on technical leadership through architecture reviews, code reviews, troubleshooting, and engineering mentorship.
Partner with cross-functional teams to mature successful R&D initiatives into enterprise-grade production solutions.
Communicate technical decisions, trade-offs, risks, dependencies, and roadmap recommendations to both technical and business stakeholders.
Drive adoption of modern data engineering practices, platform automation, and platform reliability disciplines.

Required qualifications

Typically 12-18 years of overall career relevant experience, including significant ownership of complex enterprise data engineering solutions.
Extensive experience designing, building, and operating large-scale production data platforms.
Expert programming skills in Python, PySpark, and modern software engineering practices.
Advanced SQL expertise, including data modeling, performance tuning, query optimization, and large-scale analytical processing.
Strong hands-on experience with distributed data processing technologies such as Apache Spark and modern lakehouse platforms.
Experience building and operating cloud-native data platforms in AWS, or Azure, or other enterprise cloud environments.
Strong knowledge of modern data architecture patterns, including data lakes, lakehouses, data meshes, data products, and event-driven architectures.
Experience with cloud storage, data integration, streaming, serverless, observability, and security services across public cloud platforms.
Experience building scalable batch and real-time data pipelines.
Experience with workflow orchestration platforms such as Apache Airflow, Databricks Workflows, AWS Step Functions, Azure Data Factory, or similar technologies.
Experience implementing CI/CD, automated testing, version control, Infrastructure as Code, and platform automation.
Strong understanding of data governance, metadata management, lineage, quality frameworks, privacy controls, and access management.
Experience diagnosing and resolving complex performance, reliability, scalability, and operational challenges.
Ability to make sound architectural decisions while balancing delivery speed, innovation, maintainability, security, and cost.
Proven ability to thrive in R&D and innovation-focused environments where priorities and requirements may evolve through experimentation.
Strong communication, collaboration, technical leadership, and mentoring skills.
Bachelor's degree in Computer Science, Engineering, Information Systems, or a related technical discipline, or equivalent practical experience.

Preferred qualifications

Experience supporting machine learning, generative AI, or agentic AI systems.
Experience with MLflow or comparable ML lifecycle tooling.
Experience with Azure Machine Learning, Azure AI Foundry, or Microsoft Fabric.
Experience with Unity Catalog, Microsoft Purview, or comparable catalog and governance platforms.
Experience with Kafka or other event-streaming technologies.
Experience with containerized and cloud-native platforms, including Docker and Kubernetes.
Experience with Terraform or another Infrastructure as Code framework.
Experience integrating structured, semi-structured, unstructured, and streaming data.
Knowledge of responsible AI, model evaluation, and AI platform observability.
Experience in payments, financial services, or another regulated industry.

For full details about this role, click “Apply on company site” to view the employer’s original posting.

Skills for this role

PythonPySparkSQLApache SparkAWSApache AirflowDatabricks WorkflowsAWS Step FunctionsAzure Data FactoryMLflowAzure Machine LearningAzure AI FoundryMicrosoft FabricUnity CatalogMicrosoft PurviewKafkaDockerKubernetesTerraform

Checking your profile…

Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.

Understand the role

What matters most

Architecting Scalable Data Platforms for Advanced AI Workloads
Design and build robust batch, streaming, and lakehouse data platform architectures that seamlessly support machine learning, generative AI, and agentic AI initiatives as specified in the Mastercard Foundry R&D goals.
Enabling the Complete AI and ML Lifecycle
Establish reliable feature engineering, training, evaluation, deployment, and inference pipelines utilizing frameworks like Apache Spark, Python, and PySpark to accelerate data science experimentation.
Embedding Data Governance, Security, and Compliance
Implement rigorous data governance, quality controls, lineage, metadata management, and privacy-by-design standards across sensitive and regulated corporate environments.
Providing Hands-on Technical Leadership and Mentorship
Lead architecture and code reviews, establish engineering standards, and mentor cross-functional teams to mature R&D concepts into enterprise-grade production solutions.

Likely challenges to think through

Balancing R&D Agility with Production Rigidity
The trade-off between moving fast with exploratory AI prototypes and enforcing enterprise-grade security, governance, and reliability standards. Prepare by demonstrating experience with incremental delivery plans and building scalable systems without introducing unnecessary platform complexity.
Enabling AI Teams Without Owning Model Research
Navigating the boundary of supporting data scientists and AI engineers without taking direct ownership of model research or data science responsibilities. Prepare by highlighting a track record of building reusable self-service data products and API capabilities that empower cross-functional teams.

Practice your answers

Technical practice

Designing Governed Lakehouse Architectures for AI Workloads

What to coverDiscuss how you would design a modern lakehouse architecture on AWS or cloud-native environments to support both batch analytics and generative AI/agentic workloads. Address data storage tiers, ACID transaction guarantees, and how metadata catalogs integrate to maintain governance and lineage.

Optimizing Distributed Processing in PySpark and Apache Spark

What to coverExplain strategies for diagnosing and resolving performance bottlenecks in large-scale PySpark data pipelines, such as data skew, excessive shuffle operations, and memory spills. Emphasize efficient data modeling and query optimization techniques.

Orchestrating Complex AI and ML Pipelines with Airflow

What to coverDescribe how you configure and manage workflow orchestration using Apache Airflow for multi-stage pipelines involving data ingestion, feature engineering, model training, and evaluation. Address dependency management, error handling, and task retries.

Implementing Data Governance, Lineage, and Access Control

What to coverExplain your approach to embedding data quality frameworks, lineage tracking, and privacy-by-design access controls into distributed data platforms. Discuss how to balance strict compliance requirements in regulated financial environments with developer velocity.

Enabling the ML Lifecycle with MLflow and Feature Stores

What to coverHow do you build and operate robust feature engineering, model training, and inference pipelines that integrate with ML lifecycle tooling like MLflow? Discuss reproducibility, model tracking, and serving reusable feature sets.

Streaming and Real-Time Event-Driven Data Ingestion

What to coverCompare batch processing with event-driven or streaming architectures (such as using Kafka) for real-time ingestion and enrichment pipelines. Address latency trade-offs, state management, and stream-processing reliability.

Infrastructure as Code and CI/CD Automation for Data Platforms

What to coverDiscuss your experience implementing automated testing, continuous integration and deployment (CI/CD), and Infrastructure as Code (e.g., Terraform) for cloud-native data platforms. Focus on ensuring reproducibility and reducing deployment risks.

Translating Ambiguous R&D Objectives into Scalable Architectures

What to coverHow do you approach translating loosely defined research or innovation objectives into practical, incremental data architecture delivery plans? Provide examples of validating architectural choices through prototyping and proof-of-concepts.

Behavioural practice

Balancing R&D Experimentation with Production Rigidity

What to coverDescribe a situation where you had to transition an exploratory R&D initiative or prototype into an enterprise-grade, secure production system. Explain how you managed scope, addressed technical debt, and aligned cross-functional stakeholders on delivery milestones using a situation-action-result format.

Influencing Cross-Functional Teams Without Direct Authority

What to coverProvide an example of how you influenced AI engineers, data scientists, and product managers to adopt modern data engineering standards and governance practices without having formal people-management authority. Highlight the communication strategies and trade-offs discussed.

Resolving Complex Production Performance and Scalability Bottlenecks

What to coverTell us about a time when a critical large-scale data pipeline suffered from severe performance degradation or reliability issues in production. Detail your diagnostic process, how you resolved the root cause under pressure, and what preventative measures you implemented.

Navigating Technical Trade-offs Between Delivery Speed and Cost Efficiency

What to coverShare an experience where you had to make a difficult architectural trade-off balancing speed of innovation, system maintainability, security, and cloud compute costs. Describe how you evaluated the options and communicated your recommendation to senior leadership.

Sharpen your knowledge

Topics to prepare

Modern Data Architecture Patterns
Conceptualize and compare data lakes, lakehouses, data meshes, and event-driven architectures. Exercise: Sketch an end-to-end lakehouse architecture supporting real-time streaming ingestion alongside batch AI training data. Relevance: Directly tied to designing governed lakehouse and modern data platform architectures for Mastercard Foundry R&D.
Data Governance and Privacy-by-Design Principles
Study metadata management, data lineage, quality controls, access management, and regulatory compliance frameworks. Exercise: Outline a framework for tracking data lineage and access control in sensitive financial environments. Relevance: Essential for implementing data governance capabilities and applying security-by-design principles.
ML Lifecycle and Feature Engineering Platforms
Understand the requirements for supporting machine learning, generative AI, and agentic AI pipelines. Exercise: Map out the lifecycle flow from raw feature engineering to model evaluation and inference deployment. Relevance: Directly required for enabling the full AI and machine learning lifecycle and supporting advanced AI workloads.

Skills to revise

Distributed Data Processing with PySpark and Apache Spark
Master large-scale data transformation, performance tuning, and optimization. Exercise: Write an optimized PySpark script to process and join large distributed datasets with minimal shuffle overhead. Relevance: Required for expert programming skills in Python and PySpark and handling distributed processing frameworks.
Workflow Orchestration and Pipeline Automation
Build and manage complex Directed Acyclic Graphs (DAGs) using Apache Airflow or similar orchestration tools. Exercise: Construct an Airflow DAG with error-handling, task retries, and dynamic task mapping for an AI data pipeline. Relevance: Directly aligned with experience requirements for workflow orchestration platforms like Apache Airflow.

Your preparation checklist

Prepare Enterprise Lakehouse Architecture Walkthrough
Develop a comprehensive architectural diagram and presentation illustrating an end-to-end lakehouse platform built on AWS or similar cloud environments. Completion means having a structured walkthrough covering data ingestion, distributed storage tiers, and ACID transactions supporting both batch analytics and generative AI workloads.
Demonstrate Distributed Data Processing Expertise
Prepare code samples and optimization case studies using Python, PySpark, and Apache Spark. Completion involves showcasing specific techniques for resolving data skew, tuning query performance, and scaling distributed data transformations.
Build an Orchestrated Data Pipeline Prototype
Create a functional prototype using Apache Airflow or comparable workflow orchestration tools. Completion includes a documented DAG demonstrating error handling, task retries, dynamic task mapping, and integration with cloud storage.
Outline a Governance and Privacy-by-Design Framework
Draft a reference framework detailing data lineage, metadata management, quality controls, and access management. Completion means being able to articulate how security and privacy principles are embedded in sensitive corporate environments.
Formulate AI Lifecycle and Feature Engineering Examples
Prepare concrete examples of supporting machine learning, generative AI, or agentic AI workloads using tools like MLflow or feature stores. Completion includes explaining how data engineering capabilities accelerate data science experimentation and model deployment.
Draft Strategic Questions for the Hiring Team
Formulate thoughtful, role-specific questions regarding Mastercard Foundry R&D priorities, collaboration models between data engineers and AI researchers, and platform scaling challenges. Completion means having 4-5 targeted questions that reflect a deep understanding of the position.

Availability last checked: 4 Nov 2026. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy

KEEP EXPLORING

Similar AI jobs

Related skills and specializations in India. Matched to this role, not your profile.

Explore more jobs

Finding similar opportunities…