New opportunity

Python / LLM Engineer (Mid-Senior)

Narix Labs · Jaipur, India

About this role

Own the AI capability of a production coaching platform, including Bispy, an LLM assistant that must follow a client's behavioural-science methodology rather than give generic advice. The role spans the coaching assistant, Qdrant retrieval, evaluation, and the assessment scoring and ML pipeline.

Responsibilities

Own prompts, context assembly, retrieval, fallbacks, and cost control for the LLM coaching layer.
Build and tune Qdrant retrieval using chunking, embeddings, and hybrid search; measure relevance and address retrieval failures.
Maintain golden datasets, automated scoring, and regression gates to catch quality changes before release.
Develop explainable assessment scoring and ML results in Python. Apply minimum-necessary handling to sensitive assessment data.

Requirements

3+ years of production Python experience and 1+ year building LLM-backed features that shipped.
Hands-on retrieval or RAG experience with vector stores and embeddings, including their failure modes; ability to explain how to detect quality regressions automatically.
Familiarity with Claude, GPT, Gemini, and open-weight model APIs, and the ability to choose among them empirically. Strong written communication and documentation of model behavior.

Preferred

Classical ML fundamentals at a scikit-learn level; experience measurably constraining LLMs to a domain methodology or brand voice; voice or real-time AI exposure. The listed stack includes Python, LLM APIs, Qdrant, RAG, evaluation tools, and PostgreSQL.

The role is full-time and based on-site in Jaipur, India. Although another line calls it remote-friendly, the posting explicitly says all roles are on-site only. The team works async-first on IST with deliberate US/EU overlap. Compensation is described as competitive and reviewed on outcomes; no amount is provided.

For full details about this role, click “Apply on company site” to view the employer’s original posting.

Skills for this role

PythonLarge language modelsLLM APIsQdrantRAGVector databasesEmbeddingsHybrid searchPrompt engineeringContext engineeringModel evaluationAutomated scoringRegression testingMachine learningscikit-learnPostgreSQLClaudeGPTGeminiWritten communication

Checking your profile…

Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.

Understand the role

What matters most

Methodology-bound coaching
The role owns Bispy’s prompts, context assembly, retrieval and fallbacks. The assistant must follow the client’s behavioural-science methodology rather than drift into generic advice.
Measurable Qdrant retrieval
Build and tune chunking, embeddings and hybrid search in Qdrant; measure relevance and identify retrieval failures, not just whether a result was returned.
Release-gating evaluation
Maintain golden datasets, automated scoring and regression gates on every change so coaching-quality regressions are caught before users encounter them.
Explainable assessment results and data boundaries
Develop the assessment scoring and ML pipeline in Python, explain results to non-engineers and give sensitive assessment data minimum-necessary treatment.
Empirical model choices and documentation
Choose among Claude, GPT, Gemini and open-weight models empirically while managing coaching-layer costs and documenting model behavior.

Likely challenges to think through

Useful coaching without methodology drift
Preparation hypothesis: a broadly helpful answer may still violate the client’s methodology. Prepare to explain how you would define observable adherence criteria, assemble supporting context and handle cases where the assistant lacks enough grounding.
Retrieval gains versus operating cost
Preparation hypothesis: richer chunking, hybrid search or more retrieved context may improve relevance while raising cost or introducing distracting material. Prepare a comparison plan using relevance measures, failure examples and downstream answer quality.
Reliable regression signals for variable outputs
Preparation hypothesis: LLM responses can vary even when quality has not meaningfully changed, making release gates difficult to calibrate. Prepare to describe representative golden cases, automated scoring, thresholds and review of disputed failures.
Explainability without unnecessary data exposure
Preparation hypothesis: explaining assessment scores may require evidence, but exposing raw sensitive inputs conflicts with minimum-necessary handling. Prepare an approach to communicating the basis of a result while limiting the assessment data used or displayed.

Practice your answers

Technical practice

Suggested practice: How would you keep Bispy's coaching responses within a client's behavioural-science methodology?

What to coverDiscuss how prompts, approved context and response checks could reinforce the methodology rather than produce generic advice. Explain how you would measure adherence and handle a request the available methodology does not cover.

Suggested practice: How would you assemble context and design fallbacks for the coaching assistant?

What to coverExplain what context a response needs, how you would limit irrelevant material, and what should happen when retrieval is weak or the model cannot give a grounded answer. Address the trade-off between response quality, latency and cost.

Suggested practice: How would you choose chunks and embeddings for Qdrant retrieval?

What to coverDescribe how chunk boundaries and embedding choices affect whether the right methodology material is retrieved. Explain how you would test changes against representative queries rather than relying on a few examples.

Suggested practice: When would hybrid search improve retrieval, and how would you diagnose failures?

What to coverCompare semantic and keyword matching for queries that use different wording or require exact terms. Discuss relevance measures and how you would distinguish a missing source, poor ranking and an answer-generation failure.

Suggested practice: What would you put in a golden dataset for the coaching assistant?

What to coverCover representative coaching requests, methodology-boundary cases and retrieval failures, with clear expectations for acceptable responses. Explain how you would keep examples useful as the system changes without masking regressions.

Suggested practice: How would you build an automated quality gate for an LLM change?

What to coverDescribe scoring for groundedness, methodology adherence and retrieval relevance, including where human review remains necessary. Explain thresholds, noisy scores and how a failed gate would inform a release decision.

Suggested practice: How would you choose among Claude, GPT, Gemini and open-weight model APIs?

What to coverPropose a comparison on the role's actual coaching tasks using quality measures alongside latency and cost. Explain how you would document observed behaviour and revisit the choice if results change.

Suggested practice: How would you make assessment scoring results explainable to non-engineers?

What to coverExplain how a Python scoring pipeline could make inputs, transformations and score interpretation traceable. If using a classical ML model, discuss the trade-off between predictive performance and explanations that remain accurate and understandable.

Suggested practice: How would you enforce minimum-necessary handling of sensitive assessment data?

What to coverDiscuss which data each scoring or LLM step actually needs and how to avoid passing unnecessary assessment details through prompts, retrieval and evaluation records. Distinguish structural data boundaries from relying on instructions to the model.

Suggested practice: How would you maintain a production Python pipeline across LLM and assessment changes?

What to coverDescribe tests and documentation that would make scoring and model-behaviour changes reviewable. Explain how you would detect an unexpected change before release and trace it to the affected pipeline stage.

Behavioural practice

Suggested practice: Tell me about taking ownership of a shipped AI feature.

What to coverUse a real situation to explain your scope, the decisions you made and how you handled competing quality and delivery needs. State the result and what evidence showed whether the feature worked.

Suggested practice: Describe a time you investigated a retrieval or model-quality failure.

What to coverSet out the symptom, your diagnostic steps and how you separated possible causes. Describe your action, the result and how you checked that the fix held.

Suggested practice: Tell me about a time evaluation evidence challenged a planned release.

What to coverDescribe the quality signal, the decision you faced and how you communicated the trade-off. Explain what you did, the outcome and what changed in the evaluation approach.

Suggested practice: Describe explaining an assessment or model result to a non-engineer.

What to coverGive the situation and what the audience needed to understand, including any uncertainty or limitation. Explain how you communicated it and what resulted from that conversation.

Suggested practice: Tell me about coordinating technical work through written, asynchronous communication.

What to coverUse a real example to show what you documented, how you resolved questions or disagreements, and what action followed. State the result and how you knew others could act on the information.

Sharpen your knowledge

Topics to prepare

Methodology-grounded coaching
Review how prompts, retrieved context and fallbacks can keep an assistant within a defined methodology. Exercise: write hypothetical methodology rules, then classify sample replies as compliant, unsupported or generic. Bispy must follow a client's behavioural-science methodology; the posting does not disclose its rules.
Retrieval failure modes
Revisit how chunking, embeddings and hybrid search affect relevance. Exercise: sketch a test set with ambiguous queries and missing-context cases, then explain how each failure would be detected. The role requires hands-on RAG experience and measurable Qdrant retrieval quality.
Evaluation and regression design
Review golden datasets, automated scoring and release gates. Exercise: define separate checks for retrieval relevance, methodology adherence and fallback behavior, including what would block a change. The posting calls for regressions to be caught automatically before release.
Empirical model selection
Compare model choices by task quality, latency and cost rather than reputation. Exercise: design a controlled comparison of Claude, GPT, Gemini and open-weight options using the same coaching cases. The role calls for empirical API selection and cost control, not allegiance to one provider.
Explainable scoring and data boundaries
Review how assessment results can be explained to non-engineers while limiting exposure of sensitive inputs. Exercise: outline a scoring result, its plain-language explanation and the minimum data each processing step needs. Both explainability and minimum-necessary handling are explicit responsibilities; scikit-learn-level ML is only preferred.

Skills to revise

Implement a coaching request flow
In Python, build a small illustrative flow that assembles context, applies hypothetical methodology rules and returns a defined fallback when evidence is insufficient. Explain where API usage and cost are measured. This practices the end-to-end LLM coaching-layer ownership described in the posting.
Tune Qdrant retrieval
Using sample, nonsensitive text, create alternative chunking and embedding configurations in Qdrant, test hybrid search and inspect misses against labeled queries. This directly practices the advertised retrieval and relevance work.
Build an evaluation gate
Create a small golden set with expected retrieval and response behaviors; run automated scores before and after a deliberate prompt or retrieval change, and make the check fail on a defined regression. This mirrors the posting's evaluation-suite responsibility.
Prototype assessment scoring
Write a Python scoring function for invented assessment inputs, test edge cases and produce a short, non-technical explanation of each result without retaining unnecessary inputs. This practices explainable scoring and sensitive-data boundaries; the example scoring rules would be illustrative, not the client's.
Document model behavior
Write a concise comparison record for the same test cases across candidate model APIs, noting observed failures, quality, cost and the reason for a choice. The posting explicitly expects empirical selection and written documentation of model behavior.

Your preparation checklist

Draft a role-specific cover note
The posting says the cover note and work links matter more than CV formatting. Prepare a concise note pointing to verifiable examples of production Python and shipped LLM-backed features; do not claim experience you cannot substantiate.
Rehearse a production project walkthrough
Prepare one project narrative covering your responsibilities, architecture, deployment, a failure you addressed, and measured results. Use it for the advertised portfolio deep-dive or practical round.
Sketch a methodology-bound coaching assistant
As a practice exercise, diagram prompts, context assembly, retrieval, fallbacks, and cost controls for an assistant that follows a client's behavioural-science methodology rather than offering generic advice. Mark assumptions because the methodology is not provided.
Prepare a retrieval investigation
Walk through how you would test chunking, embeddings, and hybrid search in Qdrant or a similar vector store. Bring a concrete example of a retrieval failure, a relevance measure, and how you would diagnose it.
Design an evaluation gate
Draft a small golden dataset outline, automated scoring criteria, and a pass/fail regression rule. Explain how the gate would catch a change in coaching quality or methodology adherence before release.
Explain assessment scoring and data boundaries
Prepare a Python scoring or classical-ML example with an explanation a non-engineer could understand. Show how you would limit sensitive assessment data to the minimum necessary; treat specific data flows as questions, not known facts.
Document a model-selection comparison
Sketch an empirical comparison of Claude, GPT, Gemini, and open-weight options using task quality, cost, and failure cases. Bring a short example of how you document model behavior and limitations.
Write questions for the employer
Ask how methodology adherence and retrieval relevance are judged, what the current evaluation gates cover, and where ownership of assessment scoring sits. Confirm the on-site Jaipur expectation, since the page also contains conflicting remote-friendly wording.

Availability last checked: 4 Nov 2026. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy

KEEP EXPLORING

Similar AI jobs

Related skills and specializations in India. Matched to this role, not your profile.

Explore more jobs

Finding similar opportunities…