New opportunity

Senior AI/ML Engineer

Ciklum · Chennai, Pune, India

About this role

Ciklum seeks a full-time Senior AI/ML Engineer for its cross-functional team in Chennai or Pune, India, with a hybrid working arrangement. The role focuses on building scalable enterprise AI applications using large language models (LLMs), retrieval-augmented generation (RAG), and NLP, while improving retrieval quality and establishing reliable evaluation practices.

Responsibilities

Design, develop, and deploy production-grade Python AI/ML applications, turning architectural designs into maintainable code. Lead technical reviews and improve performance, scalability, and reliability.
Audit RAG pipelines and retrieval architectures. Monitor KPIs for pgvector-based systems, optimize hybrid retrieval using BM25, vector databases, and semantic search, and improve relevance, accuracy, and user experience.
Build automated evaluation frameworks using RAGAS and other LLM assessment methods. Create Python validation tools to measure faithfulness, relevance, answer correctness, and retrieval effectiveness; establish quality scorecards and benchmarks; and incorporate metrics into CI/CD pipelines and Azure DevOps release quality gates.
Guide mid-level engineers, review code, mentor team members, and work with product managers, architects, and stakeholders on technical roadmaps and delivery plans.

Requirements

A bachelor's or master's degree in Computer Science, Artificial Intelligence, Machine Learning, Data Science, or a related field, plus at least 6 years of experience in machine learning, NLP, generative AI, or related domains.
Strong Python and software engineering skills; hands-on RAG experience; understanding of BM25, vector databases, semantic search, embeddings, pgvector, and retrieval optimization.
Expertise in LLM evaluation frameworks such as RAGAS; experience building automated AI testing and validation frameworks; knowledge of Azure DevOps, CI/CD, and quality gates; and experience conducting code reviews and mentoring engineers.

Desirable: Experience with Azure OpenAI, OpenAI, Anthropic, or other enterprise LLM platforms; LangChain, LlamaIndex, or Semantic Kernel; PostgreSQL and large-scale data pipelines; observability, monitoring, and AI governance; or cloud deployment on Azure, AWS, or GCP.

Benefits mentioned include learning events, Udemy access, language courses, company-paid certifications and medical insurance, mental health support, and financial and legal consultations.

For full details about this role, click “Apply on company site” to view the employer’s original posting.

Skills for this role

PythonMachine LearningNatural Language ProcessingGenerative AILarge Language ModelsRetrieval-Augmented GenerationBM25Vector DatabasesSemantic SearchpgvectorEmbeddingsRetrieval OptimizationRAGASLLM EvaluationAutomated TestingAzure DevOpsCI/CDCode ReviewPostgreSQLAzure OpenAIOpenAIAnthropicLangChainLlamaIndexSemantic KernelAzureAWSGoogle Cloud PlatformTechnical LeadershipMentoring

Checking your profile…

Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.

Understand the role

What matters most

Production AI engineering
Design, develop, and deploy Python LLM, RAG, and NLP applications; turn architectural designs into maintainable code and improve scalability and reliability.
RAG retrieval quality
Audit retrieval pipelines, monitor KPIs in pgvector-based systems, and optimize hybrid BM25, vector, and semantic search for relevance, accuracy, and user experience.
Automated AI evaluation
Build RAGAS-based and other validation tools for faithfulness, relevance, answer correctness, and retrieval effectiveness; establish benchmarks and connect metrics to CI/CD and Azure DevOps quality gates.
Technical leadership and delivery
Lead technical reviews, guide and mentor mid-level engineers, and work with product managers, architects, and stakeholders on roadmaps and delivery plans.

Likely challenges to think through

Balancing retrieval gains with system cost
A likely challenge is improving relevance without excessive latency or complexity across BM25, vector, and semantic retrieval. Prepare to explain how you would audit a pgvector-based pipeline, select KPIs, compare retrieval approaches, and validate a change against user experience.
Making evaluation useful for releases
A likely difficulty is turning imperfect LLM evaluation signals into dependable release decisions. Prepare to discuss test-set design, how faithfulness and answer correctness differ from retrieval effectiveness, and how benchmarks and quality gates could detect regressions without relying on one score.
Aligning engineering quality with delivery
Technical reviews, mentoring, and roadmap collaboration can create trade-offs between maintainability and delivery pace. Prepare an example of how you would communicate a technical risk, agree on priorities with stakeholders, and guide implementation through code review.

Practice your answers

Technical practice

How would you turn an AI solution blueprint into a production-grade Python application?

What to coverThe role involves translating architectural designs into scalable, maintainable Python code. Preparation suggestion: Explain how you would structure components, test changes, and review performance and reliability before deployment.

How would you audit an existing RAG pipeline?

What to coverThe posting calls for leading technical audits of RAG pipelines and retrieval architectures. Preparation suggestion: Walk through how you would trace documents from ingestion to answer, identify failure points, and prioritize changes using evidence.

Which KPIs would you monitor for a pgvector-based retrieval system?

What to coverThe role specifically calls for instrumenting and monitoring pgvector retrieval KPIs. Preparation suggestion: Discuss measures of retrieval effectiveness and latency, how you would segment results, and what would trigger investigation.

When would you combine BM25 with vector and semantic retrieval?

What to coverOptimizing hybrid retrieval across BM25, vector databases, and semantic search is a stated responsibility. Preparation suggestion: Compare lexical and semantic strengths, then explain how you would test whether combining them improves relevance without unacceptable latency.

How would you troubleshoot a drop in retrieved-context relevance?

What to coverThe posting emphasizes improving retrieval relevance, accuracy, and user experience. Preparation suggestion: Describe how you would inspect queries, embeddings, candidate results, and ranking, then validate a proposed fix against representative cases.

How would you evaluate faithfulness and answer correctness in a RAG application?

What to coverThe role calls for RAGAS and other LLM assessment methods, including measures of faithfulness and answer correctness. Preparation suggestion: Distinguish groundedness in retrieved context from correctness against an expected answer, and explain how you would review questionable scores.

How would you build a Python validation harness and AI quality scorecard?

What to coverBuilding automated validation tools, benchmarks, and quality scorecards is part of the role. Preparation suggestion: Explain how you would maintain representative test cases, record retrieval and answer metrics, and compare changes against a baseline.

How would you use evaluation results in Azure DevOps release quality gates?

What to coverThe posting explicitly calls for integrating evaluation metrics into CI/CD pipelines and Azure DevOps quality gates. Preparation suggestion: Describe a repeatable evaluation run, a justified pass/fail threshold, and how you would handle noisy results before blocking a release.

How would you improve an AI application's scalability without losing reliability?

What to coverThe role includes continuous improvement of model performance, scalability, and reliability. Preparation suggestion: Explain how you would locate bottlenecks, compare optimization trade-offs, and verify that retrieval and answer quality remain stable.

How would you assess whether an AI orchestration framework is useful for this solution?

What to coverKnowledge of LangChain, LlamaIndex, or Semantic Kernel is listed as desirable, not required. Preparation suggestion: Compare an appropriate framework with simpler Python components in terms of maintainability, observability, and control over the RAG pipeline.

Behavioural practice

Tell me about a time you guided an engineer through an AI/ML engineering challenge.

What to coverThe role includes day-to-day guidance and mentoring of mid-level engineers. Preparation suggestion: Use situation, action, and result to show how you supported their reasoning and what changed afterward.

Describe a code review in which you raised an important quality concern.

What to coverLeading technical reviews and conducting code reviews are stated responsibilities. Preparation suggestion: Explain the situation, how you communicated and resolved the concern, and the resulting effect on code quality or delivery.

Tell me about a time you aligned technical work with product and architecture stakeholders.

What to coverThe role involves collaborating on technical roadmaps and delivery plans. Preparation suggestion: Describe the situation, the trade-offs you communicated, the actions agreed upon, and the result.

Describe a time you led a technical audit that changed a team's priorities.

What to coverLeading audits of RAG pipelines and retrieval architectures is part of the advertised work. Preparation suggestion: Give a real situation, explain how you gathered and presented evidence, and state what the team did as a result.

Tell me about a time you improved an engineering or testing practice across a team.

What to coverThe posting calls for continuous improvement across AI engineering and testing practices. Preparation suggestion: Outline the initial problem, how you gained adoption for a change, and the measurable or observable result.

Sharpen your knowledge

Topics to prepare

RAG architecture and failure modes
The role requires hands-on RAG experience and technical audits. Revise by diagramming a retrieval-to-answer pipeline and explaining where irrelevant context, missing evidence, or unsupported answers can arise.
Hybrid retrieval and ranking
The posting names BM25, vector databases, embeddings, and semantic search. Revise by comparing what lexical and vector retrieval each capture, then explain how you would diagnose a query for which one approach performs poorly.
pgvector retrieval performance
The role calls for monitoring KPIs in pgvector-based systems and improving relevance and user experience. Revise by outlining which retrieval-quality and performance measures you would inspect during an audit, and how you would distinguish a retrieval problem from an answer-generation problem.
LLM evaluation design
The posting explicitly calls for RAGAS and measures of faithfulness, relevance, answer correctness, and retrieval effectiveness. Revise by defining each measure, identifying the evidence it needs, and explaining why a single aggregate score could hide a regression.
Release quality and technical leadership
The role includes Azure DevOps quality gates, code reviews, and mentoring. Revise by explaining how evaluation results could inform a release decision and how you would communicate a failing result to engineers and stakeholders.

Skills to revise

Audit a RAG pipeline
Practice tracing a sample query through retrieval and answer generation; record failure points and propose a prioritized fix. This mirrors the advertised responsibility to lead RAG architecture audits.
Tune hybrid search
Practice comparing BM25 and vector results for a small set of queries, inspecting missed relevant passages, and adjusting a combined ranking approach. This targets the posting’s explicit hybrid-retrieval optimization work.
Build a Python validation harness
Practice writing a small Python harness that runs a fixed set of questions, captures retrieved passages and answers, and reports faithfulness, relevance, correctness, and retrieval measures. The posting asks for automated AI validation and RAGAS-based evaluation.
Define a release quality gate
Practice turning evaluation results into a scorecard with a documented pass/fail rule and a regression report suitable for a CI/CD run. This reflects the stated Azure DevOps quality-gate responsibility; the exact thresholds are a preparation exercise, not a stated employer standard.
Review and explain AI/ML code
Practice reviewing a Python RAG change for maintainability, tests, performance, and reliability, then write actionable feedback and explain the trade-offs to a mid-level engineer. The role explicitly includes technical reviews and mentoring.

Your preparation checklist

Map your evidence to the requirements
Prepare a brief evidence sheet covering the stated degree and 6+ years of relevant experience, plus Python, RAG, retrieval optimization, LLM evaluation, Azure DevOps, code reviews, and mentoring. Note where you would need to explain a gap rather than imply experience.
Build a production RAG walkthrough
Prepare one architecture diagram and a concise walkthrough of a Python AI application you have built, if available. Explain how you translated a design into maintainable code and addressed deployment, scalability, and reliability.
Rehearse a retrieval audit
Work through an example pgvector-based RAG pipeline. Produce an audit plan that identifies retrieval KPIs, compares BM25, vector, and hybrid search, and shows how you would investigate weak relevance or accuracy.
Draft an evaluation scorecard
Prepare a small example of a Python validation harness using RAGAS or another LLM evaluation method. Show how it would measure faithfulness, relevance, answer correctness, and retrieval effectiveness, with a benchmark and scorecard.
Sketch a release-quality design
Draw how automated evaluations could run in CI/CD and inform an Azure DevOps quality gate. Explain your proposed handling of failed checks and changing benchmarks; present thresholds as design choices, not Ciklum policy.
Prepare leadership examples
Select examples, if available, of a technical review, code review, mentoring decision, and collaboration with product managers or architects. For each, outline the decision, trade-offs, and outcome.
Write questions for the team
Ask which retrieval KPIs and evaluation benchmarks matter most today, how release quality gates are used, and where this engineer would lead audits or guide mid-level engineers. These questions target responsibilities stated in the posting.

Availability last checked: 5 Nov 2026. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy

KEEP EXPLORING

Similar AI jobs

Related skills and specializations in India. Matched to this role, not your profile.

Explore more jobs

Finding similar opportunities…