AI Platform Senior Engineer, QA

What matters most

• Building the Enterprise AI Trust & Validation Framework

  The core mandate is to establish a robust framework for validating non-deterministic AI outputs. You must design strategies that ensure reliability, transparency, and scalability across complex agentic workflows and multi-agent orchestration layers.

• Advanced AI Model and Agent Evaluation

  You are responsible for implementing sophisticated validation techniques, including LLM-as-Judge, semantic similarity, and statistical testing. This requires moving beyond traditional software testing to evaluate groundedness, coherence, hallucination, and output drift in AI-generated narratives.

• Automation Framework Architecture

  You must design and maintain scalable automation frameworks using Python, JavaScript/TypeScript, Pytest, and Playwright. This includes integrating these frameworks into CI/CD pipelines to support continuous regression testing for both traditional APIs and complex AI pipelines.

• Performance Engineering for AI Workloads

  The role requires identifying bottlenecks in AI-heavy systems through load and stress testing. You will need to monitor system performance and scalability risks to ensure the platform can handle high-stakes, judgement-heavy tasks efficiently.

Likely challenges to think through

• Validating Non-Deterministic AI Outputs

  Unlike traditional software, AI outputs are inherently non-deterministic, making standard pass/fail assertions difficult. Prepare by demonstrating how you define 'success' for subjective outputs and how you use statistical methods or LLM-based evaluation to quantify quality.

• Testing Multi-Agent Orchestration

  Testing complex, multi-agent workflows involves managing dependencies between different AI agents. Prepare by discussing your approach to isolating components within an agentic system and how you validate the 'reasoning' or 'decision-making' steps rather than just the final output.

• Balancing Speed with Rigorous Validation

  In a fast-paced AI environment, there is a tension between rapid deployment and the need for deep validation. Prepare by explaining how you prioritize test coverage for high-risk areas while maintaining velocity in CI/CD pipelines.

Topics to prepare

• Non-deterministic AI System Testing

  Focus on strategies for validating systems where outputs vary, such as LLM-based agents. Exercise: Define a test plan for a RAG-based chatbot that measures groundedness and hallucination rates using a golden dataset. This is critical for the role's focus on the Enterprise AI Trust & Validation Framework.

• LLM-as-a-Judge Evaluation

  Understand the methodology of using a stronger LLM to evaluate the outputs of smaller models. Exercise: Research and draft a Python script using an LLM to score the coherence and relevance of another model's response. This directly addresses the requirement for implementing advanced AI evaluation techniques.

• Agentic Workflow Orchestration

  Study how multi-agent systems interact and maintain state across complex tasks. Exercise: Map out a test strategy for a multi-agent pipeline where one agent generates a narrative and another validates it for factual accuracy. This is essential for testing the company's agent-driven platforms.

Skills to revise

• Playwright for AI Automation

  Develop proficiency in using Playwright to automate UI interactions for AI-driven chat interfaces. Exercise: Create a test script that simulates a user conversation with an AI agent, capturing and asserting the correctness of the generated text. This is a core requirement for building E2E automation.

• Performance Testing for AI Workloads

  Learn to measure system latency and throughput under high-concurrency AI requests. Exercise: Use K6 to simulate multiple concurrent API calls to an inference endpoint and identify potential bottlenecks in response time. This aligns with the role's responsibility for performance engineering.

• Python-based Test Frameworks

  Master Pytest for building modular and reusable test suites. Exercise: Refactor a set of functional tests into a Pytest-based framework that includes fixtures for environment setup and teardown. This is necessary for maintaining the automation frameworks mentioned in the job description.

• Develop an AI Evaluation Framework Strategy

  Draft a high-level approach for validating non-deterministic AI outputs, specifically focusing on how you would measure groundedness, coherence, and hallucination rates in multi-agent workflows using techniques like LLM-as-Judge.

• Prepare a Technical Walkthrough of Automation Architecture

  Create a portfolio summary or presentation detailing a past project where you built an E2E automation framework using Python or TypeScript/Playwright, highlighting how you integrated it into CI/CD pipelines.

• Demonstrate Performance Engineering for AI Workloads

  Prepare a case study on how you have conducted load and stress testing for API-heavy or AI-driven systems, specifically mentioning tools like K6 and how you identified and resolved system bottlenecks.

• Formulate Targeted Questions for the Engineering Team

  Prepare questions regarding Accrete's current AI observability stack, how they manage drift in production agents, and the specific challenges they face in validating their Knowledge Engine's ground truth.