Databricks AI & Data Engineer

The Databricks AI & Data Engineer designs and deploys agentic AI solutions that automate data governance, data quality, metadata management, lineage analysis, policy enforcement and stewardship across enterprise data platforms. The full-time role is based in Hyderabad, IN-TG, India, with a 40-hour workweek.

Responsibilities

• Build scalable AI agents using large language models, retrieval-augmented generation and orchestration frameworks. Develop autonomous and human-in-the-loop workflows for data cataloging, business glossaries, compliance validation, data ownership, issue remediation and governance orchestration.
• Integrate agents with data catalogs, metadata repositories, data quality and master data management tools, cloud data platforms, and governance technologies including Ataccama, Collibra, Informatica, Microsoft Purview, Databricks, Snowflake, AWS and Azure.
• Work with data governance engineers, data engineers, architects and business stakeholders to identify use cases, define agent capabilities and align solutions with enterprise data strategy. Create proof-of-value demonstrations and pilots, prioritize development backlogs with delivery teams and clients, and incrementally onboard governance controls.
• Implement monitoring, observability, guardrails and controls for security, compliance, transparency and responsible AI. Support proposals, architecture discussions, effort estimates and client presentations; contribute reusable frameworks, accelerators and best practices.

Required qualifications

• At least 8 years in data management, data engineering, data governance, metadata management or enterprise data architecture; at least 5 years designing and implementing enterprise governance, MDM, metadata, data quality or regulatory compliance solutions.
• At least 3 years developing AI/ML, generative AI or agentic AI solutions with LLMs and orchestration frameworks; at least 3 years of hands-on Python and modern AI framework experience, such as LangChain, LangGraph, Semantic Kernel, AutoGen, CrewAI, OpenAI, Azure AI Foundry or Amazon Bedrock.
• At least 3 years with Azure, AWS or Google Cloud Platform; at least 2 years integrating governance platforms such as Collibra or Databricks Unity Catalog; and at least 2 years designing APIs, workflow orchestration, event-driven architectures or enterprise integrations.
• Experience with governance policies, lineage, data quality controls, stewardship, compliance monitoring and Agile delivery. Strong client-facing communication skills for workshops, requirements gathering, architecture reviews and executive demonstrations.