Databricks AI & Data Engineer

The Databricks AI & Data Engineer designs, develops and deploys agentic AI solutions that automate data governance, quality, metadata management, lineage analysis, policy enforcement and stewardship across enterprise data platforms. The full-time role is based in Hyderabad, India, with 40 hours per week stated.

Responsibilities

• Build scalable AI agents using large language models (LLMs), retrieval-augmented generation (RAG) and orchestration frameworks. Develop autonomous and human-in-the-loop support for data cataloging, business glossaries, policy validation, data ownership, issue remediation and governance workflows.
• Integrate agents with data catalogs, metadata repositories, data quality and master data management platforms, cloud data platforms and governance technologies. Create proof-of-value demonstrations and pilots for governance automation.
• Work with governance engineers, data engineers, architects, delivery teams and client stakeholders to identify use cases, define agent capabilities, align architecture with enterprise data strategy and prioritize development backlogs.
• Implement monitoring, observability, guardrails and governance controls for security, compliance, transparency and responsible AI. Support proposals, solution architecture discussions, effort estimates and client presentations; contribute reusable frameworks and best practices.

Required qualifications

• At least 8 years in data management, data engineering, data governance, metadata management or enterprise data architecture; at least 5 years designing or implementing enterprise governance, master data management, metadata, data quality or regulatory compliance solutions.
• At least 3 years developing AI/ML, generative AI or agentic AI solutions with LLMs and orchestration frameworks; at least 3 years of hands-on Python and modern AI frameworks such as LangChain, LangGraph, Semantic Kernel, AutoGen, CrewAI, OpenAI, Azure AI Foundry or Amazon Bedrock.
• At least 3 years with Azure, AWS or Google Cloud Platform; at least 2 years integrating governance platforms such as Collibra or Databricks Unity Catalog; and at least 2 years designing APIs, workflow orchestration, event-driven architectures or enterprise integrations.
• Experience with governance policies, lineage, quality controls, stewardship, compliance monitoring and Agile delivery. Strong client-facing communication skills for workshops, requirements gathering, architecture reviews and executive demonstrations.