New opportunity

Agent Engineer - Evals and Harness

AryaXAI · Mumbai, India

About this role

Build the shared execution harness and evaluation system for autonomous agents used in software engineering, data science and AI research. The systems are designed to run in customer environments. This is a systems and infrastructure role in which concurrency, state and measurement matter more than prompting; the harness and evaluation work are equally important.

Harness responsibilities:

Develop the core agent loop and execution model, including orchestration, concurrency, sub-agent coordination, cancellation, timeouts, checkpointing and resume.
Manage context and state for long-running tasks; define shared tool protocols, schemas, versioning and libraries used by agent teams.
Build reproducible, snapshot-capable, resource-bounded sandboxes and establish retry, partial-failure and idempotency behavior. Define agent traces for debugging, training signals and customer audit records.

Evaluation responsibilities:

Build task suites from real work for each agent type and infrastructure to run them in parallel at scale. Develop deterministic verifiers and model-graded rubrics, and calibrate graders.
Run regression gates on harness changes, assessing quality alongside cost and latency. Apply seeds, variance, pass@k and confidence to determine whether improvements are meaningful.
Convert production failures into permanent tests and build tools for trace inspection, replay and comparison. Work with agent and research teams on evaluation design, post-training and interpretation of agent behavior.

Required qualifications

Advanced Python and strong software engineering fundamentals, including experience shipping and operating production services. Candidates should understand concurrency and distributed systems; containers, Docker and OCI internals; resource isolation; test and CI infrastructure; statistical measurement; and observability through distributed tracing and structured logging. Systematic debugging, comfort with loosely specified problems and ownership of production behavior are important. The team is hiring across experience levels, including early-career engineers with strong fundamentals.

Preferred qualifications

AI-system evaluation or benchmarking experience is a strong plus. Agentic systems, LLM inference and serving, developer tooling, and readable open-source contributions are also pluses. The posting lists Mumbai and remote work, with applicant eligibility specified as India.

For full details about this role, click “Apply on company site” to view the employer’s original posting.

Skills for this role

PythonSoftware engineeringDistributed systemsConcurrencyAsync executionWorker poolsQueuesDockerOCIContainerizationSandboxingTest harnessesCIStatistical analysisDistributed tracingStructured loggingAgentic systemsAI evaluationBenchmarkingLLM inferenceLLM servingDeveloper tooling

Checking your profile…

Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.

Understand the role

What matters most

Reliable shared agent execution
The posting centers the role on a shared loop for three agent types, including orchestration, sub-agent coordination, cancellation, timeouts, checkpointing and resume. Prepare to explain how an execution model handles interrupted or partially completed work.
Consistent execution environments and interfaces
The role owns reproducible, snapshot-capable, resource-bounded sandboxes, plus shared tool schemas and versioned libraries. Prepare to discuss how those contracts stay consistent across repository tasks, training runs and data analysis in customer environments.
Useful state and traces
Long-running context requires retention and compaction; agent traces must support debugging, training signals and customer audit records. Prepare to explain what state and trace data would make a run understandable and replayable.
Credible evaluations grounded in real work
The posting calls for task suites for each agent type, parallel execution, deterministic checks where possible and calibrated model-graded rubrics otherwise. Prepare to distinguish what can be verified directly from what needs judgment.
Evidence-based regression decisions
Harness changes face regression gates that consider quality, cost and latency; results must account for seeds, variance, pass@k and confidence. Production failures should become permanent tests. Prepare to explain how you would avoid calling a noisy result an improvement.

Likely challenges to think through

Recovery without duplicate or lost work
Likely difficulty, not a confirmed problem: concurrency, retries and resume can conflict when a step fails halfway through. Prepare a concrete failure sequence and explain the trade-off between retrying, restoring a checkpoint and stopping for handoff.
One substrate for different workloads
Likely difficulty, not a confirmed problem: a shared harness must serve coding, data science and AI engineering agents while sandboxes run in customer environments. Prepare to discuss which tool and execution contracts should be common and where workload-specific behavior should remain configurable.
Realistic evaluations that yield clear signals
Likely difficulty, not a confirmed problem: real-work tasks and model-graded outcomes may be variable, while large parallel suites consume time and resources. Prepare to discuss repeat runs, grader calibration and how much evidence would justify a quality gain given its cost and latency.

Practice your answers

Technical practice

Suggested practice: How would you structure the shared agent loop to coordinate concurrent steps and sub-agents?

What to coverThe role describes a loop that plans, acts, observes, recovers and stops. Explain how you would bound concurrency, propagate cancellation and timeouts, and decide when work can safely continue.

Suggested practice: How would you manage context and resume a long-running agent task?

What to coverDiscuss what state must survive a checkpoint and how retention or compaction could preserve information the agent needs later. Address the trade-off between a smaller context and an accurate account of completed work.

Suggested practice: How would you design a shared tool protocol used by different agent teams?

What to coverCover schemas, versioning and internal libraries that make tool calls consistent across agent types. Explain how you would evolve a schema without silently changing the meaning of existing calls.

Suggested practice: How would you build a reproducible, resource-bounded execution sandbox?

What to coverThe posting calls for snapshot-capable sandboxes that support repository tasks, training runs and data analysis. Discuss container and OCI behavior, resource isolation, and how you would investigate a task that succeeds locally but fails in the sandbox.

Suggested practice: What should happen when an agent step fails halfway through?

What to coverDistinguish retryable errors from failures that should stop the task and hand control back. Explain how idempotency and partial-failure handling affect retries, especially when a step has already produced side effects.

Suggested practice: What would an agent trace need to capture?

What to coverThe role calls for one trace schema serving debugging, training signals and customer audit records. Describe how structured events and distributed tracing could reconstruct decisions and tool outcomes without losing the sequence of a run.

Suggested practice: How would you assemble and run evaluation suites from real work?

What to coverExplain how tasks for coding, data science and AI engineering agents could reflect their distinct work while using shared evaluation infrastructure. Address parallel execution, reproducibility and isolation between runs.

Suggested practice: When would you use deterministic verification rather than a model-graded rubric?

What to coverGive criteria for choosing objective checks where possible and rubrics where outcomes require judgment. Explain how you would calibrate a grader and examine disagreement or inconsistent scores.

Suggested practice: How would you decide whether a harness change improves performance?

What to coverThe posting calls for regression gates that consider quality, cost and latency, using seeds, variance, pass@k and confidence. Discuss how repeated runs and uncertainty would inform a decision rather than treating a single benchmark result as conclusive.

Suggested practice: How would you turn an intermittent production failure into a lasting regression test?

What to coverDescribe how trace inspection, replay and run-to-run comparison could identify the failure mechanism. Explain how you would make the resulting test useful in parallel CI while investigating, rather than ignoring, flakiness.

Behavioural practice

Suggested practice: Tell me about a loosely specified infrastructure problem you took ownership of.

What to coverUse a real situation, the decisions and actions you personally took, and the result. Focus on how you established scope and verified production behavior rather than assuming ownership ended at implementation.

Suggested practice: Describe a difficult production failure you debugged systematically.

What to coverSet out the symptoms and available evidence, then explain how your actions distinguished the cause from plausible alternatives. Close with the observed result and any test or monitoring change that helped prevent a repeat.

Suggested practice: Describe a time you worked across teams to define a shared technical interface.

What to coverGive the situation, your actions to reconcile different needs, and the result for the teams involved. A tool protocol or shared infrastructure example is particularly relevant to the cross-agent work in this role.

Suggested practice: Tell me about a time measurement challenged an apparent improvement.

What to coverExplain the initial claim, how you checked the evidence or variability, and what you did next. State the result, including any trade-off among quality, cost or latency if it was relevant.

Suggested practice: Describe how you resolved persistent test flakiness.

What to coverUse a real situation, the investigation and corrective actions you took, and the resulting effect on the test suite. Focus on how you established the cause rather than simply rerunning or suppressing failures.

Sharpen your knowledge

Topics to prepare

Concurrency and failure semantics
The posting calls for async execution, worker pools, queues, backpressure, cancellation, timeouts, retries and idempotency. As a revision exercise, draw a sub-agent workflow and explain what should happen if a step fails halfway through or is cancelled.
Long-running state and tool contracts
The role covers checkpointing, resume, context retention and compaction, plus shared tool schemas and versioning. As a revision exercise, specify what state a task must save to resume safely and how a tool-schema change could affect an older checkpoint.
Reproducible sandbox execution
The posting specifies Docker and OCI internals, resource isolation, snapshots and resource-bounded execution in customer environments. As a revision exercise, reason through why the same task might pass locally but fail in a sandbox, then outline how to reproduce the failure.
Evaluation design and scoring
The role uses real-work task suites, deterministic verifiers where possible, and calibrated model-graded rubrics otherwise. As a revision exercise, choose a repository task and define an objective pass check, a rubric for what the check misses, and a way to inspect grader disagreements.
Statistical interpretation of regressions
The posting names seeds, variance, pass@k, confidence, and quality alongside cost and latency. As a revision exercise, compare two small sets of repeated-run results and explain what further runs would be needed before claiming an improvement.
Traces and permanent failure tests
The role calls for distributed tracing, structured logging, trace replay and comparison, and turning production failures into tests. As a revision exercise, map the events needed to reconstruct one failed agent run and identify the assertion that would catch it again.

Skills to revise

Implement a Python agent loop
Practice suggestion: build a small Python plan-act-observe loop with concurrent tasks, bounded work, cancellation and timeouts; test a sub-agent failure. This exercises the advanced Python and execution-model work explicitly described in the posting.
Test checkpoint and retry behavior
Practice suggestion: save and restore task state, then inject a failure after an action but before its result is recorded. Verify whether a retry duplicates work. This mirrors the posting’s checkpoint, resume, partial-failure and idempotency responsibilities.
Diagnose a Docker sandbox mismatch
Practice suggestion: run the same small task locally and in a resource-limited Docker container, then document any dependency, filesystem or limit-related difference. Docker, resource isolation and reproducible sandboxes are explicit requirements; this is a preparation exercise, not a stated interview task.
Build a parallel evaluation suite
Practice suggestion: run several representative tasks in parallel, record each result, and add a regression check that reports failures, cost and latency. This reflects the posting’s test-harness, CI and at-scale evaluation work.
Score and compare repeated runs
Practice suggestion: score a task with a deterministic check, draft a rubric for a less objective outcome, and compare repeated results by seed, variance and pass@k. The posting explicitly requires verifier design, grader calibration and measurement literacy.
Inspect and replay a failure trace
Practice suggestion: emit structured events for actions, tool calls, retries and stopping; replay one failed run and compare it with a successful run. This targets the trace inspection, systematic debugging and observability work in the posting.

Your preparation checklist

Prepare a production-system walkthrough
The application asks about a complex AI or ML platform system, including scale, workload, infrastructure and direct contribution. If you have a relevant example, prepare a one-page account of its architecture, operating constraints and your ownership; otherwise, do not invent one.
Sketch the shared agent execution loop
The role covers planning, acting, observing, recovery and stopping across three agent types. As a preparation exercise, draw a Python-oriented execution model showing concurrency, sub-agent coordination, cancellation, timeouts, checkpointing and resume.
Work through a sandbox and failure scenario
The posting calls for reproducible, snapshot-capable, resource-bounded execution in customer environments. Prepare a short design explaining isolation, tool-schema versioning, retries, idempotency and what happens when a step fails halfway through.
Draft an evaluation and regression plan
The role gives harness and evals equal weight. Outline a real-work task suite, parallel runs, deterministic checks versus model-graded rubrics, grader calibration and a gate that considers quality, cost and latency.
Show how you would judge a measured improvement
Prepare a small worked example using seeds, variance, pass@k and confidence to explain how many runs you would need and when you would withhold a claim of improvement. This is a preparation exercise, not a stated interview task.
Prepare a debugging and observability example
The application asks for a hard production failure. If you have one, document the symptoms, trace and structured-log evidence, root cause, fix and permanent regression test. Be ready to explain how replay or run comparison would help investigate an agent failure.
Write focused questions for the team
Ask how the three agent teams share the harness, what constraints customer environments impose, how production failures enter eval suites, and how research and engineering decide whether a regression gate should pass.

Availability last checked: 5 Nov 2026. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy

KEEP EXPLORING

Similar AI jobs

Related skills and specializations in India. Matched to this role, not your profile.

Explore more jobs

Finding similar opportunities…