New opportunity

AI Harness Engineering PM

A65 · Singapore

About this role

The AI Harness Engineering PM role at A65 is a unique position at the intersection of product management and agentic engineering. You will define, build, and own the harness layer—the infrastructure, connectors, evaluation systems, and feedback loops—that enables AI agents to function reliably within A65's advertising products. This is a greenfield function requiring a blend of technical depth and product instinct.

Key Responsibilities:

Design and own the agent harness framework, including execution environments, orchestration scaffolding, and evaluation pipelines.
Build and maintain integrations with internal systems like Slack, Hubspot, and Google Drive, ensuring reliability and security.
Develop evaluation and benchmarking systems to measure agent performance, reliability, and safety.
Act as a primary signal source by engaging users and using real-world tasks to drive harness improvements.
Collaborate with ML researchers to co-evolve the harness and define the agent product roadmap.
Troubleshoot agent failures, hallucinations, and orchestration errors while contributing to architecture and code reviews.

Required Qualifications

4+ years of software engineering experience, preferably in full-stack or platform engineering.
2+ years of product thinking experience.
Hands-on experience with testing frameworks, evaluation systems, or developer tooling.
Strong grasp of CI/CD, source control, and Agile practices.
Familiarity with LLM-based systems, AI agents, and autonomous workflows, including specific knowledge of agent loops, tool use, reasoning, and context engineering.
Deep familiarity with agent products like Claude Code, Cursor, or GitHub Copilot.
Ability to operate as a sole contributor in a greenfield environment.
Strong UI/UX design sensibility and English communication skills.

Preferred Qualifications

Experience with AI agent frameworks (LangChain, LangGraph, CrewAI, AutoGen).
Knowledge of distributed systems, Kubernetes, cloud infrastructure, or observability tooling.
Experience building simulation environments or synthetic test datasets.
Open-source community involvement or experience leading a product roadmap as a sole PM.

For full details about this role, click “Apply on company site” to view the employer’s original posting.

Skills for this role

Software EngineeringFull-stack EngineeringPlatform EngineeringTesting FrameworksEvaluation SystemsDeveloper ToolingCI/CDSource ControlAgileLLM APIKV CacheAgent LoopTool UseReasoningPlanningMCPMemorySubagentMulti-AgentPrompt EngineeringContext EngineeringProduct ManagementLangChainLangGraphCrewAIAutoGenKubernetesCloud InfrastructureObservability

Checking your profile…

Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.

Understand the role

What matters most

Architecting the Agent Harness Framework
As a greenfield role, the primary priority is designing the infrastructure layer that sits between the LLM and the user. This includes building robust execution environments, orchestration scaffolding, and reliable connectors to internal systems like Slack and Hubspot. You must define the technical standards for how agents interact with the environment.
Establishing Evaluation and Benchmarking Systems
You are responsible for defining what constitutes agent success. This requires building pipelines to measure performance, reliability, and safety. You must move beyond simple testing to create simulation environments or synthetic datasets that quantify agent behavior in production-like scenarios.
Translating User Signal into Product Roadmap
Acting as the primary signal source, you must bridge the gap between real-world agent tasks and technical improvements. This involves extracting actionable data from user interactions and agent failures to prioritize the roadmap and co-evolve the harness with ML research efforts.

Likely challenges to think through

Navigating Ambiguity in a Greenfield Function
The role lacks an existing team, playbook, or inherited structure. You must balance high-level product strategy with low-level engineering tasks (code reviews, troubleshooting). Prepare by demonstrating your ability to self-start and define processes from scratch without relying on established organizational support.
Managing Agent Non-Determinism
Troubleshooting agent hallucinations and orchestration errors is inherently difficult due to the non-deterministic nature of LLMs. You will need to prepare by discussing your approach to observability, logging, and root-cause analysis in systems where traditional debugging methods may be insufficient.
Balancing Technical Depth with Product Instinct
The role requires being both a hands-on engineer and a product manager. A common challenge is over-indexing on technical implementation at the expense of user experience, or vice versa. Prepare by highlighting examples where you have successfully aligned technical architectural decisions with specific user-centric product outcomes.

Practice your answers

Technical practice

Designing Agent Harness Architecture

What to coverExplain how you would architect a harness layer that sits between an LLM and external tools. Discuss how you would handle orchestration, state management, and the integration of execution environments while ensuring the system remains modular and scalable.

Evaluation Pipelines for Agent Reliability

What to coverDescribe your approach to building an evaluation framework for autonomous agents. Focus on how you would measure performance, safety, and reliability, including the use of synthetic datasets and real-world task benchmarking.

Troubleshooting Agent Hallucinations and Errors

What to coverWhen an agent fails or produces a hallucination, what is your systematic process for root-cause analysis? Discuss how you would use observability tools and logs to distinguish between model-level issues, orchestration errors, and tool-use failures.

Managing Tool Use and Context Engineering

What to coverHow do you optimize an agent's ability to use tools effectively while managing context window constraints? Address trade-offs between prompt engineering, memory management, and the use of techniques like KV caching or sub-agent delegation.

CI/CD for Agentic Workflows

What to coverHow does CI/CD differ when deploying agentic systems compared to traditional software? Discuss how you would implement automated testing for agent behavior and ensure that updates to the model or tools do not introduce regressions.

Integrating External Systems Securely

What to coverWhen building connectors for platforms like Slack or HubSpot, how do you manage authentication, rate limiting, and data security? Explain how you ensure these integrations remain reliable as the underlying APIs evolve.

Leveraging Agent Frameworks

What to coverCompare the strengths and weaknesses of frameworks like LangGraph or CrewAI for building multi-agent systems. Discuss when you would choose a pre-built framework versus building a custom orchestration layer.

Defining Agent Performance Metrics

What to coverBeyond simple success rates, what metrics would you track to determine if an agent is genuinely helpful to a user? Explain how you would translate these metrics into actionable product roadmap items.

Behavioural practice

Operating in a Greenfield Environment

What to coverDescribe a time you were tasked with building a function from scratch without an existing team or playbook. How did you define your initial priorities, and how did you validate your approach with stakeholders?

Collaborating with ML Researchers

What to coverGive an example of how you have bridged the gap between technical research and product requirements. How do you influence model development to ensure it aligns with the practical needs of the harness and the end user?

Translating User Signals into Product Decisions

What to coverDescribe a situation where you had to prioritize features based on conflicting user feedback or raw data. How did you decide what to build next, and how did you communicate that decision to the team?

Demonstrating Ownership as a Sole Contributor

What to coverProvide an example of a project where you had to act as both the engineer and the product manager. How did you balance the need for high-quality code with the need to ship features and meet product deadlines?

Sharpen your knowledge

Topics to prepare

Agentic Harness Architecture
The role requires designing the infrastructure layer between LLMs and end-user applications. Exercise: Draft a high-level architecture diagram for an agentic system that integrates with external APIs (like Slack or Hubspot), focusing on how you would handle orchestration, state management, and error recovery when an agent fails to execute a tool correctly.
Agent Evaluation and Benchmarking
You must define metrics for agent reliability and safety. Exercise: Propose a methodology for creating a synthetic test dataset to evaluate agent performance in a specific marketing workflow, including how you would measure 'hallucination' rates versus 'successful task completion' in a simulated environment.
Product-Engineering Feedback Loops
The role acts as a bridge between user signals and technical roadmap. Exercise: Create a framework for capturing and prioritizing 'agent failure' signals from real-world usage, detailing how you would categorize these errors to inform both immediate engineering fixes and long-term model/harness improvements.

Skills to revise

Agent Framework Implementation
Practical experience with frameworks like LangGraph or AutoGen is preferred. Exercise: Build a small prototype using LangGraph that implements a multi-agent workflow where one agent performs a task and a second agent validates the output against a predefined schema.
Observability for AI Systems
Monitoring agent behavior is critical for troubleshooting. Exercise: Design an observability strategy for an agentic loop, identifying which specific events (e.g., tool call latency, context window usage, or reasoning chain depth) you would log to effectively debug orchestration errors.
Developer Tooling and CI/CD
The role requires maintaining reliable deployment pipelines. Exercise: Outline a CI/CD pipeline specifically for an AI agent product, focusing on how you would integrate automated 'evals' into the build process to prevent regressions in agent reasoning capabilities.

Your preparation checklist

Develop an Agent Harness Architecture Proposal
Draft a high-level design document for an agent harness that manages tool orchestration, execution environments, and feedback loops. Focus on how you would handle agent reliability, error logging, and state management when integrating with external APIs like Slack or Hubspot.
Design an Evaluation and Benchmarking Framework
Create a plan for measuring agent performance, specifically addressing how to detect hallucinations and orchestration failures. Include a strategy for building synthetic test datasets to validate agent reasoning and tool-use accuracy before deployment.
Prepare a Product Roadmap for a Greenfield Function
Outline a 6-month roadmap for the harness layer that balances immediate technical debt, infrastructure stability, and user-facing feature delivery. Demonstrate how you would prioritize tasks as a sole contributor while collaborating with ML researchers.
Conduct a Deep-Dive Analysis of Agentic Tools
Prepare a comparative analysis of your experience with tools like Cursor, Claude Code, or LangGraph. Be ready to discuss specific strengths and weaknesses of these frameworks in terms of developer experience, context engineering, and reliability.
Formulate Strategic Questions for the Hiring Team
Prepare questions regarding the current state of A65's agentic stack, the specific challenges they face with agent reliability in advertising products, and how they define the boundary between model research and harness engineering.

Availability last checked: 18 Oct 2026. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy

KEEP EXPLORING

Similar AI jobs

Related skills and specializations in Singapore. Matched to this role, not your profile.

Explore more jobs

Finding similar opportunities…