New opportunity

Generative AI Test Engineer - Python/Automation Testing

SRS Solutions · Chennai, India

About this role

SRS Solutions seeks a Generative AI Test Engineer to build and maintain testing frameworks for AI-powered applications. The role focuses on validating large language models (LLMs), retrieval-augmented generation (RAG) pipelines and other AI workflows for accuracy, reliability, performance and scalability. It is a full-time position available in Chennai, Pune, Hyderabad, Bengaluru, Coimbatore, Ahmedabad or Mumbai.

Responsibilities

Design end-to-end test architecture and build automated frameworks for GenAI applications, LLM-based systems and RAG pipelines.
Develop Python test suites for APIs, backend services, databases and AI components. Perform API, integration, regression and performance testing, and validate GenAI models and prompt-engineering frameworks.
Integrate automated tests into CI/CD pipelines and design scalable testing frameworks for cloud-based AI solutions.
Monitor results, investigate failures and recommend improvements while working with developers, data scientists and product teams to identify quality risks.

Required qualifications

The listing states 6–9 years of experience; the detailed requirements also specify at least 2 years testing Generative AI, AI/ML or data-intensive applications. Structured posting data lists a bachelor's degree.
Hands-on Python automation testing and experience testing LLMs and GenAI applications are mandatory. Experience with RAG, LangChain or other AI frameworks, AI evaluation tools, multiple automation tools beyond Selenium, SQL, Snowflake, API testing and backend automation is required.
Understanding of the Software Testing Life Cycle, CI/CD test integration and exposure to AWS, Azure or Google Cloud Platform are required.

Preferred qualifications include prompt-engineering validation, AI model evaluation, scalable cloud-native test architecture, and performance and security testing of AI systems. Analytical, debugging, problem-solving, communication and collaboration skills are also preferred.

For full details about this role, click “Apply on company site” to view the employer’s original posting.

Skills for this role

Generative AILLM testingRAGPythonAutomation testingAPI testingBackend testingIntegration testingRegression testingPerformance testingAI model evaluationPrompt engineering validationLangChainSeleniumSQLSnowflakeSTLCCI/CDAWSAzureGoogle Cloud PlatformSecurity testingTest architectureDebuggingProblem-solvingCommunicationCollaboration

Checking your profile…

Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.

Understand the role

What matters most

Own end-to-end GenAI test architecture
The role calls for designing and maintaining automated frameworks for LLM applications, RAG pipelines and AI workflows. Prepare to explain how you would structure reusable tests across those components, rather than only individual test cases.
Evaluate LLM and RAG quality
The posting emphasizes accuracy, reliability and performance, including validation of models and prompt-engineering frameworks. Prepare a concrete approach to evaluating generated answers and retrieval results, with clear criteria for investigating failures.
Automate across application layers
Python suites must cover APIs, backend services, databases and AI components, with integration, regression and performance testing. Prepare to describe how tests would verify behavior across these layers, including SQL and Snowflake data checks.
Make testing continuous and scalable
The role includes CI/CD integration and scalable frameworks for cloud-based AI solutions, with exposure to AWS, Azure or GCP. Prepare to explain how you would organize automated checks for routine runs and broader performance testing.
Turn failures into quality improvements
The engineer will monitor results, analyze failures and work with developers, data scientists and product teams on quality risks. Prepare an example of how you would isolate a failure, present evidence and recommend a next step.

Likely challenges to think through

Judging variable AI outputs
A likely challenge is distinguishing an acceptable answer from a regression when LLM wording varies. Prepare to discuss evaluation criteria that test factual quality and reliability without requiring every response to match one exact string.
Locating failures in a RAG workflow
A likely challenge is determining whether a poor answer stems from retrieval, prompt handling, the model or a backend dependency. Prepare to explain how component-level evidence and end-to-end tests would narrow the cause.
Balancing coverage with pipeline speed
A likely challenge is fitting API, integration, regression and performance checks into CI/CD without making feedback impractically slow. Prepare to explain which checks you would run frequently and which need broader, less frequent runs; this is a testing trade-off, not a stated SRS Solutions problem.

Practice your answers

Technical practice

How would you design an end-to-end test architecture for a GenAI application?

What to coverThe role includes owning test architecture for LLM applications, RAG pipelines and AI workflows. Suggested answer focus: explain test layers, coverage across the software testing life cycle, and how you would balance fast checks against deeper evaluations.

How would you test a RAG pipeline when an answer is inaccurate?

What to coverHands-on RAG testing is specified in the posting. Suggested answer focus: describe checks that distinguish retrieval problems from answer-generation problems, using traceable test inputs and expected evidence.

How would you evaluate an LLM application's accuracy and reliability?

What to coverThe posting calls for LLM testing and AI model evaluation. Suggested answer focus: propose representative test cases, evaluation criteria and human review where automated judgments may be insufficient.

How would you validate a prompt change without missing regressions?

What to coverValidating prompt-engineering frameworks is a stated responsibility. Suggested answer focus: compare outputs against a versioned set of cases and explain how you would handle acceptable variation rather than relying solely on exact matches.

How would you structure Python tests for APIs and backend AI components?

What to coverThe role requires Python automation across APIs, backend services and AI components. Suggested answer focus: cover reusable test setup, integration boundaries, assertions and the trade-off between isolated and end-to-end tests.

How would you verify database-dependent results using SQL and Snowflake?

What to coverThe posting specifies hands-on SQL and Snowflake experience. Suggested answer focus: explain checks for data correctness and consistency, including how you would make test data and queries repeatable.

Which automated checks would you put in CI/CD, and when would they run?

What to coverIntegrating automated tests into CI/CD is a stated responsibility. Suggested answer focus: distinguish quick regression checks from slower AI evaluations, and explain how failures should be reported for investigation.

How would you choose automation and AI evaluation tools for this test framework?

What to coverThe posting mentions AI evaluation tools, frameworks such as LangChain, and integrating automation tools beyond Selenium. Suggested answer focus: compare tools by the interfaces they test, maintainability and suitability for evaluating AI behavior; avoid assuming any particular tool is already in use.

How would you assess performance and scalability for a cloud-based AI system?

What to coverThe role includes performance testing and scalable test frameworks for cloud-based AI solutions; security testing is preferred. Suggested answer focus: discuss representative workloads, response-time and reliability measures, and how you would investigate bottlenecks without assuming a specific cloud platform.

How would you investigate an intermittent failure in an AI test suite?

What to coverMonitoring results, analyzing failures and recommending improvements are stated duties. Suggested answer focus: separate application defects from test instability and variable model outputs, then explain what evidence would support a corrective action.

Behavioural practice

Tell me about a time you worked across teams to identify a quality risk.

What to coverThe role calls for collaboration with developers, data scientists and product teams. Suggested answer focus: give the situation, your actions to establish and communicate the risk, and the resulting decision or outcome.

Describe a difficult test failure you investigated.

What to coverFailure analysis and actionable recommendations are part of the role. Suggested answer focus: recount the symptoms, how you narrowed down the cause, what you recommended and the result.

Tell me about a time you had to prioritize testing under constraints.

What to coverThe posting spans API, integration, regression, performance and AI evaluation work. Suggested answer focus: explain the competing risks, how you chose coverage and what happened; this prioritization scenario is a preparation suggestion, not a stated employer process.

Describe a time you improved an automation framework.

What to coverDesigning, developing and maintaining automated frameworks is central to the role. Suggested answer focus: give the original limitation, the changes you made and evidence of their effect on test reliability or maintainability.

Tell me about a time you communicated uncertain test results to stakeholders.

What to coverThe role combines AI evaluation, result monitoring and cross-functional communication. Suggested answer focus: describe the uncertainty, how you presented the evidence and limitations, and the outcome of the discussion.

Sharpen your knowledge

Topics to prepare

LLM quality and prompt validation
The posting calls for validating GenAI accuracy, reliability and performance; prompt-engineering validation is preferred. Revision exercise: define pass criteria for a small set of prompts covering factual correctness, inconsistent answers and changes after a prompt revision. This is a preparation exercise, not a specified employer test.
RAG failure modes
Hands-on RAG experience and testing RAG pipelines are stated requirements. Revision exercise: trace a hypothetical question through retrieval and answer generation, then identify separate checks for irrelevant retrieved content, missing evidence and an answer unsupported by the retrieved content.
STLC and end-to-end test architecture
The posting requires STLC knowledge and assigns responsibility for end-to-end GenAI test architecture. Revision exercise: sketch a test strategy that separates component, API, integration and regression coverage, with entry criteria and a way to report quality risks.
Performance and cloud scalability
Performance testing and scalable frameworks for cloud-based AI solutions appear in the responsibilities; cloud-platform exposure is required. Revision exercise: outline how you would compare response times and failure rates as concurrent request volume increases, and distinguish application failures from test-framework capacity limits. No specific cloud service is prescribed.

Skills to revise

Python automation framework design
Python automation is mandatory, and the posting asks for frameworks that integrate multiple automation tools beyond Selenium. Revision exercise: write a small Python test harness with reusable setup, assertions and result reporting; explain how another automation tool could plug in without changing test cases. The harness is a suggested exercise, not a specified project.
LLM and RAG test execution
The role requires testing LLM applications and hands-on RAG experience, and cites frameworks such as LangChain and AI evaluation tools. Revision exercise: create test inputs and expected evidence checks for a sample RAG flow, then record which failures arise in retrieval versus answer generation. The sample flow is for practice.
API and backend test automation
API testing and backend automation are required; the responsibilities include integration and regression testing. Revision exercise: draft and automate checks for successful responses, invalid inputs and downstream failures in a hypothetical AI-enabled API, then identify which should run as regression tests.
SQL and Snowflake validation
Hands-on SQL and Snowflake experience is listed as required. Revision exercise: write SQL checks for missing records, duplicates and mismatched values in a hypothetical dataset used by an AI workflow; explain how a failed database check would affect an end-to-end test.
CI/CD test integration
Integrating automated tests into CI/CD pipelines is required. Revision exercise: specify which fast API checks run on each change and which longer GenAI regression and performance checks run separately, including what results would block a release. This schedule is a preparation suggestion, not a stated company process.
Failure investigation and reporting
The responsibilities include monitoring results, analyzing failures and recommending improvements with developers, data scientists and product teams. Revision exercise: take a hypothetical failed RAG test, separate retrieval, model-response, backend and test-harness causes, then write a concise defect report with evidence and a recommended next check.

Your preparation checklist

Map the requirements to evidence
Create a one-page matrix for the posting’s Python automation, LLM and RAG testing, AI evaluation tools, automation beyond Selenium, SQL and Snowflake, STLC, CI/CD, and cloud exposure. Note a concrete example where available; leave gaps explicit rather than implying experience.
Prepare a GenAI testing walkthrough
Outline one relevant project, if available: the system under test, your test architecture, evaluation approach, failures found, and recommendations made with developers or data scientists. If using a practice scenario instead, label it hypothetical.
Build a small Python test example
Prepare code or pseudocode showing API and backend assertions, database checks, and an AI-component regression test. Be ready to explain test data, failure reporting, and any automation tools used beyond Selenium.
Practice an LLM and RAG evaluation exercise
Draft a test plan with sample queries, retrieval and answer checks, prompt variations, and proposed measures for accuracy and reliability. Explain how you would investigate a failed result; present the measures as your design choices, not SRS Solutions’ standards.
Sketch a scalable test architecture
Draw a proposed framework for integration, regression, and performance tests, including CI/CD execution, result monitoring, and failure triage in an AWS, Azure, or GCP setting. Identify trade-offs rather than assuming the employer’s stack.
Prepare a quality-risk discussion
Write up one debugging example or hypothetical failure investigation with evidence, root-cause checks, a recommendation, and how you would communicate it to product and engineering teams. Include performance or security considerations where relevant; the posting lists security testing as preferred.
Draft focused questions for SRS Solutions
Ask which LLM and RAG workflows need coverage, how AI evaluation and Snowflake are used, what CI/CD and cloud environment the team uses, and who owns quality thresholds. Clarify the experience range if needed: the page header says 6–9 years, while the body renders it as “69 Years.”

Availability last checked: 5 Nov 2026. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy

KEEP EXPLORING

Similar AI jobs

Related skills and specializations in India. Matched to this role, not your profile.

Explore more jobs

Finding similar opportunities…