AI Test Engineer - VLabs

Vialto Partners seeks an AI Test Engineer for Vialto Labs (VLabs) Quality Engineering. The role validates the performance, reliability and integrity of production AI solutions used in tax and immigration services and other operational workflows. Working with the Programme Test Manager and engineering, product and delivery teams, the engineer turns AI testing strategy into reusable evaluation frameworks and pipelines.

Responsibilities

• Design executable, adversarial and boundary tests for LLM outputs, OCR pipelines, document classification and extraction, and agentic workflows. Assess output structure, consistency, accuracy, hallucinations, misclassifications and production readiness against defined thresholds.
• Build Python evaluation frameworks, parameterized tests, scoring mechanisms and AI-as-Judge evaluations; calibrate evaluation reliability and integrate continuous testing into CI/CD pipelines.
• Monitor drift through fixed baseline datasets and scheduled re-evaluation, set degradation thresholds, and identify regressions to support release gating. Develop and update ground truth datasets with subject matter experts and define classification and extraction standards.
• Test end-to-end agentic workflows for data integrity, error propagation and fallback behavior. Test AI pipeline APIs using Python and Postman/Newman, and check data persistence and pipeline integrity with SQL. Scale reusable quality practices, uphold Responsible AI and privacy requirements, and communicate model-readiness risks and evaluation findings to stakeholders.

Requirements

• At least 7 years in software testing, including 2–3 years focused on production AI/ML-enabled systems; experience with AI evaluation strategies, ground truth datasets, drift detection, scalable evaluation pipelines and multi-step agentic workflows.
• Advanced Python; LLM evaluation tools such as deepeval, RAGAS or promptfoo; output validation, grounding checks, statistical evaluation, OCR, visual language model and document AI testing, API testing, and SQL. Familiarity with LangChain, LlamaIndex or similar frameworks. Independent execution, analytical rigor and clear communication are expected. A bachelor’s degree is required.

Preferred

Experience in regulated or compliance-driven environments, Azure AI Foundry or AWS Bedrock, and an advanced degree in Computer Science, Data Science or a related field.

Location: Bangalore office, with the possibility of hybrid work.