New opportunity

Senior Software Engineer - Semantic Data Lake

WEX · Bengaluru, India

About this role

WEX is seeking a Senior Software Engineer for the Semantic Data Team in Bengaluru to help re-imagine its enterprise data platform, transforming raw data into semantically meaningful, trusted business assets and building AI-native data platforms supporting core 360 data objects.

Responsibilities

Engineer the Operational Backbone: Design and implement the core infrastructure of the Control Plane, serving as the central execution engine that orchestrates the entire data lifecycle from ingestion and transformation to final distribution.
Shift from Reactive to Proactive: Transform platform reliability by building sophisticated observability frameworks and automated quality gates to ensure data trust.
Drive Self-Service Autonomy: Eliminate operational bottlenecks by developing intuitive portals and golden templates, empowering domain teams to autonomously build, manage, and consume trusted data products.
Champion Platform Governance: Automate critical compliance and security policies, including end-to-end data lineage, role-based access control, and PII protection.
Prepare for AI-Native Scale: Build the foundational data architecture required for our AI future, contributing to context and RAG-based systems for next-generation intelligence.
Leverage AI coding assistants like Claude, GitHub Copilot, and Cursor to accelerate development, draft transformation logic, generate tests, and produce semantic documentation.

Required qualifications

4–8 years of experience in data engineering or software engineering with a focus on data transformation, modeling, or analytics platforms.
Strong proficiency in SQL and at least one general-purpose language such as Python or Scala.
Demonstrated experience as an AI-native engineer using tools like Claude, GitHub Copilot, or Cursor.
Hands-on experience with LLM-driven architectures, designing RAG systems, building agentic workflows such as LangGraph and CrewAI, and working with vector or graph databases.
Deep expertise in building and managing complex orchestration systems specifically Airflow and quality-first pipelines using Great Expectations.
Fluency in the modern DevOps stack including Terraform, Kubernetes, and CI/CD automation.
Proven track record of implementing automated governance utilizing DataHub or similar platforms.
Solid understanding of data quality practices including validation, enrichment, schema enforcement, and business rule encoding.

For full details about this role, click “Apply on company site” to view the employer’s original posting.

Skills for this role

SQLPythonScalaLangGraphCrewAIAirflowGreat ExpectationsTerraformKubernetesCI/CDDataHub

Checking your profile…

Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.

Understand the role

What matters most

Control Plane Architecture & Pipeline Orchestration
Design and implement the core operational backbone and control plane infrastructure to orchestrate the entire data lifecycle from ingestion to distribution. This requires deep expertise in managing complex data orchestration systems using Airflow, building robust transformation logic in SQL, Python, or Scala, and deploying scalable infrastructure using Terraform and Kubernetes.
AI-Native Integration & Agentic Workflow Design
Embed AI-driven capabilities into the data platform by designing RAG systems, building agentic workflows using frameworks like LangGraph and CrewAI, and utilizing vector or graph databases. Integrate AI coding assistants such as Claude, GitHub Copilot, and Cursor into daily workflows while critically reviewing generated code for correctness.
Data Governance, Security & Lineage Automation
Implement automated compliance and governance frameworks, specifically utilizing platforms like DataHub to manage end-to-end data lineage, role-based access control (RBAC), and PII protection across core enterprise data assets.
Proactive Observability & Data Quality Engineering
Move from reactive firefighting to proactive platform reliability by establishing automated quality gates and data quality validation frameworks using Great Expectations, alongside comprehensive observability monitoring.

Likely challenges to think through

Balancing AI-Driven Speed with Enterprise Governance
Using AI coding assistants and building agentic workflows accelerates development, but maintaining strict compliance, data lineage, and semantic correctness across sensitive business domains (customer, fleet, health, payments) creates a tension between velocity and rigorous validation. Prepare by establishing clear Spec-Driven Development standards and rigorous code review guidelines for AI-generated artifacts.
Scaling Self-Service Autonomy Across Domain Teams
Designing intuitive portals and golden templates that allow diverse domain teams to build and manage data products independently can lead to fragmentation or inconsistent data quality if guardrails are not robust. Prepare by focusing heavily on schema enforcement, automated quality checks with Great Expectations, and structured metadata cataloging via DataHub.

Practice your answers

Technical practice

Designing a Control Plane for Data Lifecycles

What to coverDiscuss how you would design the core infrastructure of an operational control plane to orchestrate ingestion, transformation, and distribution across enterprise data lakes. Focus on how you ensure idempotency, state management, and seamless recovery from partial ingestion failures.

Implementing Quality-First Pipelines with Great Expectations

What to coverExplain how you integrate data quality validation frameworks like Great Expectations directly into data pipelines. Address how you handle schema enforcement, automated quality gates, and preventing downstream consumption of unverified or corrupted data assets.

Orchestrating Complex Workflows in Apache Airflow

What to coverDescribe your approach to managing complex dependency trees and task scheduling in Airflow at scale. Discuss strategies for handling dynamic task generation, resource allocation, and optimizing execution time across large multi-tier data pipelines.

Architecting RAG Systems and Agentic Workflows

What to coverExplain how you would design an agentic workflow using frameworks like LangGraph or CrewAI combined with vector or graph databases to manage context and ontologies. Address how you maintain semantic accuracy and handle retrieval bottlenecks.

Automated Governance and Lineage Tracking with DataHub

What to coverDiscuss how you implement automated compliance, role-based access control (RBAC), and end-to-end data lineage tracking using DataHub or similar metadata platforms. Focus on making security an inherent feature of core data objects.

Infrastructure as Code and Self-Healing Deployments

What to coverExplain how you use Terraform and Kubernetes to build scalable, self-healing infrastructure for data platforms. Describe how you manage environment parity, secrets rotation, and automated rollback strategies in CI/CD pipelines.

Proactive Observability and Automated Alerting

What to coverCompare building reactive logging frameworks versus proactive observability systems using tools like Grafana and alerting integrations. Highlight specific metrics and anomaly detection techniques you rely on to catch data drift and latency spikes early.

Responsible Use of AI Coding Assistants in Engineering

What to coverDescribe your workflow for utilizing AI coding assistants like Claude, GitHub Copilot, or Cursor for drafting transformation logic and tests. Emphasize how you apply Spec-Driven Development and critically review AI-generated artifacts for business rule compliance.

Behavioural practice

Balancing Development Velocity with Rigorous Data Governance

What to coverDescribe a situation where you had to accelerate feature delivery or data pipeline creation using AI tools while ensuring strict adherence to enterprise compliance, security, and governance standards. Share how you structured your approach, handled trade-offs, and what the final outcome was.

Scaling Self-Service Autonomy for Cross-Functional Teams

What to coverGive an example of a time you built golden templates or developer tools to empower non-centralized domain teams to build and manage data products independently. Explain the challenges you faced in maintaining quality control and how you measured success.

Transitioning from Reactive Incident Firefighting to Proactive Reliability

What to coverTell me about a time you inherited a fragile data platform plagued by silent failures and reactive alerts. Describe the steps you took to introduce automated quality gates, observability frameworks, and how you drove long-term platform reliability.

Collaborating Across Diverse Technical and Business Stakeholders

What to coverDescribe a project where you translated complex business concepts, KPIs, or scoring algorithms from domain experts and data scientists into robust, scalable data models. Explain how you managed alignment and resolved conflicting technical priorities.

Sharpen your knowledge

Topics to prepare

AI-Native Data Platform Architecture & Control Planes
Revise the concepts of control plane runtime layers, managing context and ontologies, and orchestrating data lifecycles for AI agents and RAG systems. This directly maps to WEX's requirement of building an agentic control plane and semantic layer to support core data objects.
Data Governance, Lineage & Automated Compliance
Study enterprise metadata management, end-to-end data lineage, role-based access control (RBAC), and PII protection policies. This is essential for fulfilling the requirement to implement automated compliance and governance using platforms like DataHub.
Observability and Automated Quality Gates
Review frameworks for shifting data platform reliability from reactive firefighting to proactive alerting, automated quality validation, and incident management integration (such as Grafana and xMatters). This directly addresses the requirement for sophisticated observability frameworks and data trust.

Skills to revise

Apache Airflow & Pipeline Reliability
Practice building, structuring, and managing complex workflow orchestration in Airflow, integrating quality-first data validation checks using Great Expectations. This aligns with the requirement for advanced data orchestration and pipeline reliability.
Agentic Frameworks & Vector/Graph Databases
Revise hands-on implementation of RAG systems and agentic workflows using frameworks like LangGraph and CrewAI alongside vector or graph databases. This relates directly to the AI-native architecture requirements.
Infrastructure as Code & DevOps Stack (Terraform & Kubernetes)
Practice provisioning infrastructure, managing deployments, and automating CI/CD pipelines using Terraform and Kubernetes. This is required for building scalable, self-healing platform infrastructure.
AI-Assisted Development & Spec-Driven Development (SDD)
Revise workflows utilizing AI coding assistants like Claude, GitHub Copilot, and Cursor for drafting transformation logic, generating tests, and practicing Spec-Driven Development while maintaining strict output validation.

Your preparation checklist

Review Data Engineering and Transformation Core Competencies
Verify proficiency in SQL, Python, or Scala for building robust data transformations, modeling layers, and analytics platforms matching the 4-8 years experience requirement.
Build an Orchestrated Pipeline with Airflow and Great Expectations
Create a sample data orchestration project using Apache Airflow integrated with Great Expectations quality gates to demonstrate automated data validation and pipeline reliability.
Design an AI-Native RAG and Agentic Workflow Prototype
Develop a mock agentic architecture or RAG system using frameworks like LangGraph or CrewAI connected to a vector or graph database, showcasing context assembly and semantic data modeling.
Demonstrate Infrastructure as Code and Deployment Automation
Prepare code samples using Terraform, Kubernetes, and CI/CD pipelines to illustrate provisioning scalable, self-healing platform infrastructure and automated deployments.
Prepare Automated Governance and Lineage Documentation
Document a strategy for implementing automated compliance, role-based access control (RBAC), and metadata cataloging using tools like DataHub to ensure inherent data security.
Formulate Targeted Questions for the Employer
Prepare specific, insightful questions about WEX's enterprise data platform migration, their adoption of AI-native workflows, and how the Semantic Data Team collaborates with domain experts.

Availability has not yet been verified. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy

KEEP EXPLORING

Similar AI jobs

Related skills and specializations in India. Matched to this role, not your profile.

Explore more jobs

Finding similar opportunities…