New opportunity

Data Engineering Consultant

UnitedHealth Group · Hyderabad, India

About this role

Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best.

Responsibilities

Develops services, controls, and reusable patterns that enable the team to deliver value safely, quickly, and sustainably in the public cloud
Develop highly-scalable applications for the Azure cloud by building microservices, Azure functions, and patterns for public cloud enabling security and privacy at scale
Foster high-performance, collaborative technical work resulting in high-quality output
Understand new product capabilities and decompose the implementation into specific functional changes for verification
Gather and analyze data to aid in informed decision-making while providing detailed, realistic estimates
Interact skillfully with business stakeholders and third-party technical organizations
Design and develop ETL/ELT solutions on Azure Databricks, LakeBase and Spark
Develop, implement, and deploy large scale data pipelines empowering machine learning algorithms, insights generation, business intelligence dashboards, reporting and new data products
Design, build, optimize, and manage modern large-scale data pipelines ETL/ELT processing to support data integration for analytics, machine learning features and predictive modelling
Write advanced / complex SQL with performance tuning and optimization
Build AI based solutions for solving business needs, process automations and improving operational efficiency
Identify ways to improve data reliability, data integrity, system efficiency and quality
Participate in architectural evolution of data engineering patterns, frameworks, systems, and platforms including defining best practices and standards for managing data collections and integration
Mentor other data engineers and provide technical direction by teaching other data engineers how to leverage cloud data platforms

Required qualifications

Graduate degree or equivalent experience
5+ years of experience in data engineering, data integration, data modeling, data architecture, and ETL/ELT processes to provide quality data and analytics solutions
5+ years of experience in SQL with designing complex data schemas and query performance optimization
3+ years of experience in Python, Scala
3+ years of experience in Apache Spark (PySpark/Spark SQL)
3+ years with DevOps automation with Terraform
3+ years of experience with API design and lifecycle management (GraphQL, REST, etc.)
3+ years of building and deploying Cloud based solutions using - Azure Databricks with UC, Snowflake, Functions, Service Bus
Experience of building LLM integrations for workflow automation/business needs
Experience with CI/CD process and tools - GitHub Actions, GIT, Artifactory, Sonar

For full details about this role, click “Apply on company site” to view the employer’s original posting.

Skills for this role

AzureDatabricksSparkPythonScalaTerraformGraphQLRESTSnowflakeLLMGitHub ActionsSQLETL/ELT

Checking your profile…

Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.

Understand the role

What matters most

Large-Scale Data Pipeline & ETL Architecture
Design, build, optimize, and manage modern large-scale ETL/ELT data processing pipelines using Azure Databricks, LakeBase, and Apache Spark to support analytics, machine learning features, and predictive modeling.
Cloud Infrastructure & DevOps Automation
Develop scalable applications and services for the Azure cloud using microservices and Azure functions, applying Terraform for DevOps automation and ensuring security and privacy at scale.
Advanced SQL & Performance Tuning
Write advanced and complex SQL queries with a strong focus on performance tuning, optimization, and designing complex data schemas.
AI & LLM Integration Solutions
Build AI-based solutions and LLM integrations for workflow automation, business needs, process automation, and operational efficiency.
Technical Leadership & Mentorship
Mentor other data engineers, provide technical direction, define best practices and standards for data collections, and teach team members how to leverage cloud data platforms.

Likely challenges to think through

Balancing High-Performance Spark & Databricks Workloads with Cloud Cost Efficiency
The challenge lies in designing complex distributed data pipelines on Azure Databricks and Spark that process massive data volumes without incurring runaway cloud expenditure. Preparation: Review advanced Spark optimization techniques, efficient partitioning strategies, cluster sizing, and cost-monitoring practices on Azure.
Integrating LLM Workflows Securely Within Healthcare Data Constraints
Building and deploying LLM integrations for workflow automation requires handling sensitive data while maintaining strict security, privacy, and compliance standards. Preparation: Familiarize yourself with secure AI integration patterns, API lifecycle management using tools like GraphQL/REST, and data governance frameworks on cloud platforms.
Scaling Infrastructure Code and CI/CD Reliability via Terraform
Maintaining robust infrastructure-as-code automation and CI/CD pipelines across multiple tools like GitHub Actions, Terraform, and Azure requires rigorous change management. Preparation: Practice modular Terraform design, automated testing for infrastructure, and streamlined deployment practices.

Practice your answers

Technical practice

Designing Scalable ETL/ELT Pipelines on Azure Databricks

What to coverAs a suggested technical review, be prepared to discuss how you design, build, and optimize large-scale ETL/ELT pipelines using Azure Databricks, LakeBase, and Apache Spark. Emphasize your approach to handling data ingestion, partitioning strategies, and performance tuning for downstream analytics and predictive modeling.

Advanced SQL Query Performance Tuning and Schema Design

What to coverDiscuss your experience in writing advanced, complex SQL queries and optimizing query performance on large datasets. Address how you analyze execution plans, handle complex joins, and design data schemas to ensure high data reliability and integrity.

PySpark and Scala for Distributed Data Processing

What to coverExplain your methodology for leveraging Python, Scala, and Apache Spark (PySpark/Spark SQL) for distributed data transformation. Highlight how you manage memory overhead, handle skewed data partitions, and write maintainable code for production data pipelines.

Cloud Infrastructure Automation with Terraform

What to coverDescribe how you use Terraform for DevOps automation and Infrastructure-as-Code (IaC) in cloud environments. Discuss how you maintain reproducible infrastructure deployments, manage state files, and enforce security policies at scale on Azure.

API Design and Lifecycle Management

What to coverDiscuss your experience with API design and lifecycle management, including REST and GraphQL principles. Explain how you ensure secure, scalable, and version-controlled API integration for microservices and Azure Functions.

Integrating Large Language Models (LLMs) for Workflow Automation

What to coverAs an inferred preparation topic, discuss architectural patterns for building LLM integrations to automate workflows and solve business needs. Address considerations around API latency, token limits, prompt security, and connecting LLM outputs to structured data pipelines.

Data Governance and Security with Azure Databricks Unity Catalog

What to coverExplain how you implement governance, access control, and security using Azure Databricks Unity Catalog (UC) alongside Snowflake. Discuss how you manage fine-grained permissions and audit trails for sensitive data assets.

CI/CD Pipelines and Automated Quality Checks

What to coverDescribe your workflow for setting up CI/CD processes using GitHub Actions, Git, Artifactory, and Sonar. Explain how you automate code quality checks, testing, and deployment of data engineering artifacts into production.

Microservices and Serverless Architecture on Azure

What to coverDiscuss the design and deployment of cloud-native applications using Azure Functions and Service Bus. Highlight how you ensure fault tolerance, scalability, and secure communication across distributed services.

Architectural Evolution and Defining Data Engineering Standards

What to coverExplain your approach to participating in the architectural evolution of data platforms and establishing best practices. Discuss how you evaluate new framework capabilities and mentor junior engineers on cloud platform adoption.

Behavioural practice

Mentoring and Elevating Engineering Standards

What to coverDescribe a situation where you had to mentor other engineers and provide technical direction on cloud data platforms. Use the STAR method to explain how you shared your knowledge, established standards, and helped the team improve their technical output.

Collaborating Across Cross-Functional and Business Stakeholders

What to coverShare an experience where you interacted with business stakeholders and third-party technical organizations to gather requirements. Detail the situation, the actions you took to translate business needs into technical data architecture, and the resulting impact on project delivery.

Resolving a Critical Performance Bottleneck in Data Pipelines

What to coverProvide an example of a time when a large-scale data pipeline suffered from severe performance bottlenecks or data quality failures. Explain how you investigated the issue, optimized the Spark/SQL code or cloud infrastructure, and ensured long-term reliability.

Balancing Speed of Delivery with Cloud Security and Compliance

What to coverDiscuss a scenario where you had to build public cloud services or applications under tight deadlines while maintaining strict security and privacy at scale. Detail how you managed trade-offs between delivery speed and architectural integrity.

Decomposing Complex Product Capabilities into Functional Changes

What to coverDescribe a time when you had to understand new product capabilities or architectural changes and decompose them into actionable steps for your team. Outline the steps you took to verify the implementation and guide your team through the transition.

Sharpen your knowledge

Topics to prepare

Large-Scale Data Architecture and ETL/ELT Design
Conceptualize and design modern large-scale data pipelines, modeling frameworks, and integration patterns to support analytics and machine learning features. Exercise: Review architectural blueprints for ingestion frameworks on cloud data platforms and map out an optimized medallion architecture for disparate healthcare data sources. Relevance: Directly addresses the core requirement for 5+ years of experience in data engineering, data integration, data modeling, and architecture.
Cloud Infrastructure, Microservices, and Security Patterns
Understand cloud-native design principles, security, privacy at scale, and serverless/microservices architecture on Azure. Exercise: Study best practices for securing public cloud environments using Azure Functions, Service Bus, and IAM controls while maintaining high scalability. Relevance: Aligns with job responsibilities to develop scalable cloud applications and reusable patterns enabling safe and sustainable public cloud delivery.
LLM Integration and Workflow Automation
Examine conceptual patterns for integrating Large Language Models into enterprise workflows and process automation. Exercise: Review architectural patterns for connecting LLM APIs with downstream data pipelines and data stores for automated insights. Relevance: Directly supports the stated requirement for building LLM integrations for workflow automation and business needs.

Skills to revise

Apache Spark and Azure Databricks Optimization
Master PySpark, Spark SQL, and Azure Databricks with Unity Catalog (UC) for distributed data processing. Exercise: Write and performance-tune a PySpark job processing large datasets, optimizing shuffle operations, caching strategies, and table governance in Databricks UC. Relevance: Essential for designing ETL solutions on Azure Databricks and Spark as explicitly requested in the posting.
Advanced SQL and Query Performance Tuning
Develop complex SQL queries, window functions, and schema designs optimized for large-scale data systems. Exercise: Analyze execution plans for complex joins and aggregations across large schemas, refactoring queries to minimize bottlenecks and latency. Relevance: Required for writing advanced/complex SQL with performance tuning and optimization.
DevOps Automation with Terraform and CI/CD
Implement Infrastructure-as-Code (IaC) and automated deployment pipelines using Terraform and GitHub Actions. Exercise: Write Terraform modules to provision cloud infrastructure and configure a CI/CD pipeline with GitHub Actions and Sonar analysis. Relevance: Aligns with mandatory qualifications requiring 3+ years of Terraform experience and CI/CD tool proficiency.
Python and Scala Data Pipeline Development
Build robust data processing scripts and applications using Python or Scala. Exercise: Construct modular data ingestion scripts in Python/Scala with robust error handling and logging. Relevance: Fulfills the 3+ years of experience requirement in Python and Scala for data pipelines.

Your preparation checklist

Prepare an End-to-End Cloud Data Pipeline Walkthrough
Assemble a structured architectural presentation or code portfolio demonstrating a large-scale ETL/ELT pipeline built on Azure Databricks, Apache Spark, and Snowflake. Highlight data modeling decisions, partitioning strategies, and performance tuning optimizations.
Build and Document an Infrastructure-as-Code (IaC) Module
Create a reusable Terraform module that provisions cloud services (such as Azure Functions or Service Bus) with integrated CI/CD workflows using GitHub Actions. Completion is marked by a clean repository with modular configurations and automated test routines.
Develop an LLM-Integrated Workflow Automation Sample
Build a functional prototype demonstrating how large language models interact with APIs (REST/GraphQL) or data pipelines for process automation. Document the security controls, API lifecycle management, and data governance considerations used.
Draft Technical Mentorship and Leadership Examples
Prepare specific STAR-format narratives highlighting your experience mentoring other data engineers, establishing data engineering standards, and defining best practices for managing data collections and architectural evolution.
Formulate Targeted Questions for the Employer
Prepare thoughtful questions regarding the team's data architecture roadmap, how Azure Databricks Unity Catalog is utilized across data products, and how engineering standards are balanced with delivery speed in public cloud initiatives.

Availability has not yet been verified. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy

KEEP EXPLORING

Similar AI jobs

Related skills and specializations in India. Matched to this role, not your profile.

Explore more jobs

Finding similar opportunities…