New opportunity

Lead Data Engineer

NXP Semiconductors · Bangalore, India

About this role

The Lead Data Engineer will drive the performance, governance, and AI-native maturity of the enterprise Data Platform in Databricks. This role focuses on designing scalable pipelines, tuning workloads for performance and cost, and operationalizing modern data and AI capabilities on the Lakehouse architecture. Responsibilities include establishing best practices for partitioning and clustering, managing Unity Catalog for data governance, building self-healing automation using AI and agentic capabilities, and leading CI/CD workflows. The role also involves mentoring engineers, conducting design reviews, and managing vendor coordination. Required Qualifications: Bachelor's or Master's degree in Computer Science, IT, or equivalent; 6+ years of data engineering experience including 2+ years hands-on with Databricks; deep understanding of Lakehouse architecture, Delta Lake, Unity Catalog, and workflow orchestration; proven ability to tune Spark workloads at production scale; advanced Python (PySpark) and SQL skills; working knowledge of CI/CD and DevOps; and experience with observability tooling. Preferred Qualifications: Experience with Databricks-native AI capabilities and agentic frameworks; familiarity with Databricks Serverless Compute and DBSQL performance tuning; Databricks Certified Professional status; and exposure to Infrastructure-as-Code. Competencies include a performance-engineering mindset, curiosity for AI-native operations, and strong platform ownership.

For full details about this role, click “Apply on company site” to view the employer’s original posting.

Skills for this role

DatabricksData EngineeringLakehouse ArchitectureDelta LakeUnity CatalogWorkflow OrchestrationApache SparkPythonPySparkSQLCI/CDDevOpsObservabilityAI-native operationsAgentic frameworksDBSQLInfrastructure-as-Code

Checking your profile…

Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.

Understand the role

What matters most

Databricks Performance and Cost Optimization
The role requires a performance-engineering mindset to tune Spark workloads, optimize cluster sizing, and manage query performance. You must be prepared to demonstrate how you identify high-cost queries and implement table design best practices like partitioning and clustering to ensure platform efficiency.
Governance and Unity Catalog Implementation
You will be responsible for defining and operationalizing data governance standards. This involves managing access control, data lineage, and security within Unity Catalog, ensuring the platform remains compliant and organized as it scales.
AI-Native Operations and Automation
A core objective is to shift the platform toward AI-native maturity. You must be ready to discuss building self-healing automation and leveraging agentic frameworks to reduce manual intervention in data pipeline maintenance.
Platform Leadership and Mentorship
Beyond technical execution, you are expected to lead design reviews, mentor junior engineers, and influence stakeholders across teams. You must be prepared to explain how you drive technical standards and vendor coordination without having direct authority over all involved parties.

Likely challenges to think through

Balancing Innovation with Platform Stability
The push for AI-native operations and agentic frameworks may conflict with the need for high-reliability production pipelines. Prepare to discuss how you manage the risk of introducing experimental automation into stable enterprise environments.
Influencing Cross-Functional Stakeholders
You will need to partner with source teams and end users to resolve performance issues. The challenge lies in driving changes to their existing processes or codebases; prepare examples of how you have successfully influenced peers and business stakeholders to adopt new technical standards.
Managing Complex Vendor Relationships
As the owner of Databricks vendor coordination, you must manage escalations and release adoption strategies. Prepare to discuss how you balance the vendor's product roadmap with the specific, immediate needs of your internal enterprise platform.

Practice your answers

Technical practice

Optimizing Spark Workloads for Cost and Performance

What to coverExplain your methodology for identifying and resolving high-cost Spark queries in a production Databricks environment. Focus on techniques like cluster sizing, adaptive query execution, and identifying data skew or inefficient join strategies.

Delta Lake Table Design Best Practices

What to coverDiscuss how you approach partitioning and Z-Ordering/clustering to improve query performance on large-scale datasets. Address the trade-offs between write latency and read performance when designing these structures.

Implementing Governance with Unity Catalog

What to coverDescribe how you would operationalize Unity Catalog to manage data access, lineage, and security across an enterprise platform. Explain how you ensure consistent governance while maintaining developer productivity.

CI/CD and DevOps for Databricks Assets

What to coverDetail your approach to automating the deployment of Databricks notebooks, jobs, and infrastructure. Suggestion: Focus on how you integrate version control, automated testing, and environment promotion for data pipelines.

Building Self-Healing Automation

What to coverHow would you leverage Databricks-native AI or agentic frameworks to build self-healing capabilities for data pipelines? Provide examples of how you might automate monitoring or error recovery for common pipeline failures.

DBSQL Performance Tuning

What to coverExplain the specific considerations for tuning Databricks SQL (DBSQL) warehouses. Address how you manage concurrency, serverless compute settings, and query history analysis to optimize user experience.

Observability in Data Platforms

What to coverWhat metrics and observability tools do you prioritize to maintain platform reliability and performance? Suggestion: Discuss how you track long-running loads and proactively alert stakeholders before performance degrades.

Infrastructure-as-Code (IaC) in Data Engineering

What to coverDescribe your experience using IaC tools to manage data infrastructure. Explain how this practice improves consistency and scalability compared to manual configuration in a cloud-based data environment.

Behavioural practice

Influencing Stakeholders Without Direct Authority

What to coverDescribe a situation where you had to convince source teams or business stakeholders to change their data practices to improve platform performance. Focus on how you communicated the technical necessity and achieved alignment.

Mentoring and Design Reviews

What to coverShare an example of how you have mentored junior engineers or led a design review process. Highlight how you balanced providing constructive feedback with maintaining high standards for platform architecture.

Managing Vendor Relationships

What to coverDescribe a time you managed a technical escalation or release adoption strategy with a vendor like Databricks. Explain how you handled the communication and ensured the outcome met your platform's requirements.

Handling Production Incidents

What to coverProvide an example of a critical production issue you resolved in a data platform. Detail the steps you took to diagnose the root cause, implement a fix, and prevent future occurrences.

Sharpen your knowledge

Topics to prepare

Databricks Lakehouse Governance and Unity Catalog
Focus on the conceptual framework of Unity Catalog for centralized access control, data lineage, and security. Exercise: Draft a hypothetical governance model for a multi-departmental data platform, detailing how you would implement fine-grained access control and audit logging to meet enterprise standards.
Performance Engineering in Spark and Delta Lake
Understand the mechanics of Spark workload optimization, including partitioning, clustering, and file management in Delta Lake. Exercise: Review a scenario of a high-cost, long-running Spark job and outline a step-by-step diagnostic approach to identify bottlenecks, such as data skew or inefficient join strategies.
AI-Native Operations and Agentic Frameworks
Explore the integration of AI capabilities within data pipelines for self-healing and automation. Exercise: Conceptualize an automated monitoring system that uses agentic logic to detect anomalies in pipeline execution and trigger corrective actions without manual intervention.

Skills to revise

Advanced PySpark and SQL Tuning
Demonstrate proficiency in writing optimized code for production-scale data processing. Exercise: Refactor a complex SQL query or PySpark transformation to reduce execution time and resource consumption, focusing on techniques like broadcast joins and predicate pushdown.
CI/CD for Data Workloads
Apply DevOps principles to the deployment of Databricks assets. Exercise: Design a CI/CD pipeline architecture that automates the testing and deployment of Databricks notebooks and Delta tables, ensuring version control and environment consistency.
Observability and Infrastructure Monitoring
Utilize tooling to track platform health and cost. Exercise: Create a dashboarding strategy to monitor key performance indicators (KPIs) such as cluster utilization, query latency, and cost-per-job, which are critical for platform ownership.

Your preparation checklist

Prepare a Databricks Performance Tuning Case Study
Draft a technical summary of a past project where you optimized Spark workloads for cost and performance. Focus on specific actions like cluster sizing, Delta Lake table partitioning, and query optimization techniques that resulted in measurable improvements.
Develop a Governance Strategy for Unity Catalog
Outline how you would implement access control, data lineage, and security policies using Unity Catalog. Be prepared to discuss how you balance strict governance with developer productivity in an enterprise environment.
Design an AI-Native Automation Workflow
Create a conceptual design for a self-healing automation pipeline using Databricks-native AI or agentic frameworks. Explain how you would monitor for failures and trigger automated remediation to maintain platform reliability.
Formulate Questions on Platform Ownership
Prepare thoughtful questions regarding NXP's current data platform maturity, such as how they currently manage vendor escalations, their roadmap for adopting Databricks Serverless Compute, or how they balance CI/CD release cycles with production stability.
Review DevOps and Infrastructure-as-Code Practices
Refresh your knowledge on integrating CI/CD workflows with Databricks assets. Prepare to discuss how you have used Infrastructure-as-Code to manage data infrastructure and ensure consistent deployment environments.

Availability has not yet been verified. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy

KEEP EXPLORING

Similar AI jobs

Related skills and specializations in India. Matched to this role, not your profile.

Explore more jobs

Finding similar opportunities…