New opportunity

Senior Software Engineer - Search

Databricks · Bengaluru, India

About this role

The Applied AI team at Databricks is seeking a Senior Software Engineer to join the Search team in Bengaluru. This role focuses on advancing AI/ML-powered search and discovery relevance models to improve how customers find assets like tables, notebooks, and ML models on the Databricks platform. Key responsibilities include designing and implementing automated ML and NLP pipelines for query understanding, ranking, and retrieval, as well as building robust frameworks for offline and online evaluation. You will collaborate with product managers to drive technology-first initiatives and contribute to the evolution of the search product. Requirements include a BS in Computer Science (MS or PhD preferred) and at least 5 years of experience developing search relevance systems at scale. Candidates should have experience applying LLMs to search relevance and proficiency in areas such as NLP, text mining, recommendations, or conversational AI. A strong foundation in computer science and a history of contributions to open-source projects are highly valued.

For full details about this role, click “Apply on company site” to view the employer’s original posting.

Skills for this role

Search RelevanceMachine LearningNLPLLMQuery UnderstandingText MiningRecommendation SystemsPersonalizationConversational AIData PreprocessingRanking and RetrievalModel Evaluation

Checking your profile…

Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.

Understand the role

What matters most

Advancing Search Relevance and Quality
The core mandate is to improve how users discover diverse assets like tables, notebooks, and ML models. You must design and deploy sophisticated ML-based ranking and retrieval models that handle the scale and complexity of the Databricks platform.
Building Robust Evaluation Frameworks
A critical priority is establishing reliable offline and online evaluation systems. You will need to create metrics and testing pipelines that accurately measure search quality improvements to support rapid experimentation and iteration.
Integrating LLMs into Search Pipelines
Leveraging LLMs for query understanding and rewrite is a key technical focus. You are expected to apply modern generative AI techniques to enhance the search experience, moving beyond traditional keyword-based retrieval.
Cross-functional Technology Leadership
You will collaborate with product managers to define technology-first initiatives. This requires translating complex search challenges into actionable product roadmaps and driving the evolution of the search product through technical influence.

Likely challenges to think through

Balancing Model Complexity with System Performance
Implementing advanced LLM-based relevance models at scale can introduce significant latency. Prepare to discuss trade-offs between model sophistication and real-time retrieval performance, focusing on how to optimize pipelines for production environments.
Evaluating Subjective Search Quality
Defining 'relevance' for diverse assets like SQL queries and ML models is difficult. Prepare by researching methodologies for building ground-truth datasets and designing A/B testing frameworks that capture user intent effectively in a professional data platform context.

Practice your answers

Technical practice

Designing Search Relevance Pipelines

What to coverExplain how you would architect an automated ML pipeline for search relevance, specifically addressing data preprocessing, query understanding, and ranking. Focus on how you ensure scalability and maintainability when handling diverse asset types like notebooks and ML models.

LLM Integration in Search Systems

What to coverDiscuss the trade-offs of using LLMs for query understanding versus traditional NLP techniques. Describe how you would integrate an LLM into a production search pipeline while managing latency and cost constraints.

Offline vs. Online Evaluation Frameworks

What to coverDescribe the methodology for building a robust evaluation framework for search ranking. Contrast the metrics used for offline evaluation, such as NDCG or MRR, with online metrics like click-through rate or user engagement, and explain how you validate improvements.

Query Understanding and Rewriting

What to coverExplain techniques for effective query understanding, such as entity extraction, intent classification, or query expansion. Provide an example of how you would handle ambiguous user queries to improve retrieval precision.

Ranking and Retrieval at Scale

What to coverDiscuss the challenges of balancing recall and precision in a large-scale retrieval system. Explain how you would implement a multi-stage ranking architecture to efficiently surface relevant assets from a massive dataset.

Personalization in Search

What to coverHow would you incorporate user-specific context or personalization into a search relevance model without compromising privacy or system performance? Suggest approaches for leveraging user interaction history to improve search results.

Handling Diverse Data Assets

What to coverDatabricks manages various assets like tables, dashboards, and SQL queries. Explain how you would design a unified indexing strategy that allows for effective cross-asset search relevance.

Open Source Contribution Impact

What to coverSuggestion: Be prepared to discuss a specific open-source project you have contributed to, focusing on the technical challenges you solved and how your code improved the project's functionality or performance.

Behavioural practice

Collaborating with Product Management

What to coverDescribe a situation where you had to align technical search initiatives with product roadmap goals. Explain how you communicated complex ML trade-offs to non-technical stakeholders to reach a consensus.

Driving Technology-First Initiatives

What to coverProvide an example of a time you identified a technical opportunity to improve a product and successfully advocated for its implementation. Detail the steps you took to demonstrate the value of this initiative to the team.

Handling Technical Disagreements

What to coverDescribe a time when you disagreed with a team member on a technical approach for a search relevance model. How did you resolve the conflict, and what was the final outcome for the project?

Adapting to Evolving Requirements

What to coverSearch products often require rapid iteration. Share an experience where you had to pivot your technical strategy due to changing business requirements or unexpected performance results. How did you manage the transition?

Sharpen your knowledge

Topics to prepare

Search Relevance and Ranking Architectures
Focus on the end-to-end lifecycle of search relevance, specifically ranking and retrieval systems at scale. Exercise: Design a system architecture that balances candidate retrieval speed with complex re-ranking models. This is directly relevant to the role's requirement for building search systems that handle diverse assets like tables and notebooks.
Offline and Online Evaluation Frameworks
Understand the methodologies for measuring search quality, including A/B testing and offline metrics like NDCG or MRR. Exercise: Define a strategy to validate a new ranking model without negatively impacting production search performance. This addresses the explicit need for robust evaluation frameworks mentioned in the job description.
LLM Integration in Search Pipelines
Explore how Large Language Models can be utilized for query understanding, intent classification, or semantic retrieval. Exercise: Compare the trade-offs between traditional keyword-based retrieval and embedding-based retrieval using LLMs. This is essential given the requirement for experience applying LLMs to search relevance.

Skills to revise

Query Understanding and Rewrite
Develop techniques for interpreting user intent, including query expansion, synonym handling, and entity recognition. Exercise: Build a prototype pipeline that transforms ambiguous user queries into structured search parameters. This is a core responsibility for improving search discovery on the Databricks platform.
Automated ML/NLP Pipeline Engineering
Master the creation of scalable data preprocessing and model training pipelines. Exercise: Create a modular pipeline that automates the ingestion of text data and the training of a relevance model. This aligns with the requirement to enable rapid experimentation and iteration for search quality.

Your preparation checklist

Prepare a technical deep-dive on search relevance systems
Select a past project where you built or improved a search relevance system at scale. Prepare to explain your architecture, the specific challenges in query understanding or ranking, and how you measured success using offline and online evaluation metrics.
Develop a case study on LLM integration for search
Prepare a detailed walkthrough of how you have applied Large Language Models to improve search relevance. Focus on specific techniques like semantic search, query rewriting, or re-ranking, and be ready to discuss the trade-offs between latency, cost, and accuracy.
Review your open-source contributions
Identify your most significant contributions to open-source projects, particularly those related to ML, NLP, or data infrastructure. Prepare to discuss the technical impact of these contributions and how they demonstrate your ability to write production-grade code.
Formulate questions for the product and engineering team
Prepare thoughtful questions regarding Databricks' current search evaluation frameworks, the specific data challenges in indexing diverse assets like notebooks and SQL queries, and how the team balances long-term research initiatives with immediate product roadmap requirements.

Availability last checked: 18 Oct 2026. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy

KEEP EXPLORING

Similar AI jobs

Related skills and specializations in India. Matched to this role, not your profile.

Explore more jobs

Finding similar opportunities…