New opportunity

AVP/VP, AI Engineer, Technology Group

GIC Private Limited · Singapore

About this role

GIC’s AI Engineering team is building a foundational platform that enables teams across the organization to develop and deploy secure, observable, production-grade AI agents. In this core-platform role, the engineer leads the architecture and technical direction of the agent runtime: the execution layer that connects agents to tools, data, and other agents and supports reliable operation at enterprise scale.

Responsibilities

Design and evolve runtime lifecycle management, multi-step workflow orchestration, durable execution, state checkpointing, compute scheduling, and the boundaries between runtime state and durable agent memory.
Scale long-running and parallel agent workflows, addressing delegation, backpressure, rate limiting, latency, failure recovery, and graceful degradation. Make architecture decisions on event-driven communication, context engineering, multi-agent coordination, and protocols including MCP and A2A.
Build resilience and production readiness through circuit breakers, retries with jitter, dead-letter queues, idempotency, structured logging, distributed tracing, health checks, deployment safety, and operational runbooks. Collaborate with SRE and platform teams.
Evaluate and extend agent frameworks, creating abstraction layers where appropriate. Mentor engineers and set standards for system design, code review, testing, documentation, API contracts, and technical specifications.

Required qualifications

8+ years in software or platform engineering, including at least 3 years building or operating AI/ML infrastructure or agentic systems at scale. Hands-on distributed-systems expertise and production experience with multi-step or autonomous systems with complex failure modes.
Experience with durable workflow engines such as Temporal (preferred), Step Functions, or Inngest; production Python services, libraries, and SDKs; agent frameworks; and scaled LLM integration, including routing, fallbacks, structured outputs, streaming, and provider failover.
System-design leadership; familiarity with containers, orchestration, infrastructure as code, and CI/CD; and cloud-platform experience, preferably AWS.

Preferred qualifications include large-scale MCP server fleets or registries, agentic memory and context engineering, multi-agent systems, event sourcing or CQRS, framework migrations, AI observability and evaluation tools, and open-source AI/ML infrastructure contributions. The position is based in Singapore and classified as permanent. Teams generally work in the office four days per week, with flexibility to choose work-from-home days and adjust the arrangement as needed.

For full details about this role, click “Apply on company site” to view the employer’s original posting.

Skills for this role

AI agentsAgent runtime architectureDistributed systemsWorkflow orchestrationTemporalState managementFault toleranceConcurrencyPythonFastAPIgRPCLangGraphCrewAIAutoGenSemantic KernelOpenAI Agents SDKLLM integrationModel routingContext engineeringMulti-agent systemsMCPA2ADockerKubernetesAmazon EKSTerraformAWS CDKCI/CDGitHub ActionsArgo CD (ArgoCD)

Checking your profile…

Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.

Understand the role

What matters most

Own the agent runtime architecture
The role leads the execution layer for multi-step workflows, durable execution, checkpointing, lifecycle management, and compute scheduling. Runtime state and durable agent memory need clear boundaries.
Scale agent workflows reliably
The posting emphasizes long-running and parallel workflows, delegation, backpressure, rate limiting, and graceful degradation while meeting tight latency budgets.
Make production readiness foundational
Resilience and observability are expected from the start, including idempotency, retries with jitter, circuit breakers, tracing, health checks, deployment safety, and runbooks in collaboration with SRE and platform teams.
Set a pragmatic framework and integration strategy
The runtime connects agents to tools, data, and other agents and touches gateway, IAM, memory, and observability. The role evaluates frameworks and decides when an abstraction improves portability and stability.
Lead through technical standards
This AVP/VP role owns architecture decisions and is expected to mentor engineers and set standards for API contracts, system design, testing, code review, and documentation.

Likely challenges to think through

Keeping state recoverable without conflating it with memory
Preparation hypothesis: checkpointing workflow progress and retaining cross-session agent memory may demand different consistency and retention choices. Prepare to explain which state must survive a retry or restart, where it belongs, and how duplicate effects are prevented.
Balancing latency against failure protection
Preparation hypothesis: retries, rate limits, and backpressure can protect downstream services but extend a workflow's response time. Prepare a design walkthrough with latency budgets, failure scenarios, recovery paths, and criteria for graceful degradation.
Preserving portability without overbuilding
Preparation hypothesis: framework-specific features may speed delivery, while broad abstractions can add complexity. Prepare to compare adopting, extending, and wrapping an agent framework, including what the shared API must guarantee and when migration cost justifies abstraction.

Practice your answers

Technical practice

How would you design an agent runtime that can resume a multi-step workflow after a failure?

What to coverThe role owns lifecycle management, durable execution, checkpointing, and compute scheduling, with Temporal preferred among workflow engines. Suggested preparation: explain what state you would persist, how you would resume work safely, and the trade-offs between workflow-engine features and runtime-owned logic.

How would you keep long-running and parallel agent workflows responsive under load?

What to coverThe posting calls out parallelism, delegation, backpressure, rate limiting, graceful degradation, and tight latency budgets. Suggested preparation: describe how you would bound concurrent work, prioritize or shed load, and measure the effect on latency and reliability.

Where would you draw the boundary between runtime state and durable agent memory?

What to coverThe runtime must support both workflow recovery and continuity across agent sessions. Suggested preparation: distinguish execution checkpoints from information retained for future use, and discuss consistency, retention, and context-selection trade-offs.

How would you prevent retries from causing duplicate or cascading tool actions?

What to coverThe posting names circuit breakers, retries with jitter, dead-letter queues, idempotent operations, and complex failure modes. Suggested preparation: trace a failed tool call through retry and recovery, identifying where idempotency and escalation are needed.

How would you choose communication patterns for agents, tools, and other agents?

What to coverThe role involves event-driven communication, multi-agent coordination, and protocol selection, including MCP and A2A. Suggested preparation: compare direct calls and asynchronous messaging, and explain how protocol boundaries affect coordination, failure handling, and secure access.

How would you make LLM integration reliable across models or providers?

What to coverThe required experience includes model routing, fallbacks, structured-output parsing, streaming, and provider failover. Suggested preparation: explain how you would validate outputs and decide when to retry, fall back, or fail, while accounting for latency.

When would you adopt an agent framework, extend it, or place an abstraction around it?

What to coverThe role explicitly requires framework evaluation and trade-offs that avoid lock-in. Suggested preparation: compare options against runtime needs such as durable execution and observability, and describe what an abstraction should expose without hiding critical behavior.

What would you instrument to diagnose a slow or failing agent workflow in production?

What to coverThe posting calls for structured logging, distributed tracing, health checks, operational runbooks, and collaboration with SRE. Suggested preparation: outline how you would follow a workflow across model, tool, and agent steps, then use the evidence to separate latency from failure causes.

How would you design stable Python interfaces for teams building on the runtime?

What to coverThe role includes production Python services, shared libraries, SDKs, API contracts, and technical specifications. Suggested preparation: discuss interface boundaries, versioning, error semantics, and tests that protect teams when runtime behavior changes.

How would you release and operate changes to the agent runtime safely?

What to coverThe posting references containers, orchestration, infrastructure as code, CI/CD, deployment safety, and cloud platforms, with AWS preferred. Suggested preparation: describe how you would test and roll out a runtime change, monitor its effect on active workflows, and recover from a bad deployment.

Behavioural practice

Tell me about a consequential architecture decision you led amid competing technical priorities.

What to coverThe role calls for system-design leadership and clear communication of trade-offs. Suggested preparation: use a real situation, your decision-making and communication actions, and the resulting outcome; distinguish your contribution from the team's.

Describe a production failure you resolved with operations or platform colleagues.

What to coverThe role emphasizes fault recovery and collaboration with SRE and platform teams. Suggested preparation: give the situation, your actions during diagnosis and recovery, and the result, including what changed to reduce recurrence.

Give an example of how you raised engineering standards through mentoring or review.

What to coverThe posting expects mentoring and standards for design, testing, code review, and documentation. Suggested preparation: identify the situation, what you did to help others adopt a better practice, and an observable result without claiming sole credit.

Tell me about building a foundational capability when requirements were still evolving.

What to coverThis is an early-stage platform role that requires pragmatic technology choices and production readiness from the outset. Suggested preparation: describe the uncertainty, the boundaries or trade-offs you chose, and what the outcome taught you.

Describe a time you evaluated whether to keep, extend, or replace a framework.

What to coverFramework strategy and portability are stated responsibilities. Suggested preparation: set out the situation, the evidence and stakeholders behind your recommendation, your actions, and the result, including any migration or maintenance trade-offs.

Sharpen your knowledge

Topics to prepare

Agent runtime architecture
The role owns lifecycle management, orchestration, durable execution, checkpointing, and compute scheduling. As preparation, diagram an agent run from start through pause, resume, and completion; mark which state must survive a worker failure.
Distributed-systems trade-offs
The posting explicitly calls for expertise in consistency, concurrency, fault tolerance, and state management. As preparation, explain how parallel steps could race to update state and compare two ways to prevent conflicting results.
Scaling and recovery
The runtime must handle long-running workflows, delegation, backpressure, rate limits, and graceful degradation. As preparation, walk through an overloaded tool dependency and decide what to queue, retry, reject, or degrade.
State, memory, and agent coordination
The role includes boundaries between runtime state and durable agent memory, plus decisions about context engineering, multi-agent coordination, MCP, and A2A. As preparation, sketch what a delegated agent receives as context and what is retained after the workflow ends.
Framework and platform architecture
The engineer evaluates agent frameworks and decides when an abstraction layer improves portability; the runtime also connects to gateway, IAM, memory, and observability capabilities. As preparation, compare two frameworks named in the posting against a short list of runtime needs, then state where an abstraction would help or add unnecessary complexity.

Skills to revise

Implement a durable workflow
As a preparation exercise, build a small multi-step workflow with a checkpoint, a parallel step, and recovery after a simulated worker restart. Temporal is preferred in the posting; Step Functions, Inngest, or an equivalent workflow engine are also listed.
Build a production-style Python interface
As a preparation exercise, implement a small Python service and reusable client library with a clear API contract and tests for invalid requests. Production Python services, FastAPI or gRPC, shared libraries, and SDKs are explicit requirements.
Handle LLM integration failures
As a preparation exercise, prototype model routing with structured-output validation, streaming, a fallback, and provider failover; test malformed output and an unavailable provider. Each of these integration concerns appears in the must-have list.
Test resilience and observability
As a preparation exercise, inject timeouts and duplicate deliveries into a workflow; verify retry jitter, idempotency, circuit breaking, and dead-letter handling, then inspect structured logs and a trace. The posting calls for these reliability and operational patterns.
Write an architecture decision
As a preparation exercise, write a short decision record and technical specification for runtime-to-tool communication, including an API contract, latency and failure trade-offs, and a test plan. Producing defensible architecture decisions, contracts, and specifications is an explicit requirement.
Review deployment readiness
As a preparation exercise, outline a containerized service deployment with health checks, a safe rollout and rollback, and a brief operational runbook. The posting requires familiarity with orchestration, infrastructure as code, CI/CD, and cloud platforms, with AWS preferred; this outline is a suggested rehearsal, not a prescribed GIC process.

Your preparation checklist

Map your experience to the requirements
Prepare a one-page evidence map for the stated 8+ years in software or platform engineering, 3+ years in AI/ML infrastructure or agentic systems at scale, and hands-on distributed-systems, Python, workflow-engine, and cloud experience. Mark gaps rather than implying experience you do not have.
Build a production-system walkthrough
Choose an actual multi-step or autonomous system you worked on. Prepare a diagram and account of your decisions on state, concurrency, failure recovery, latency, and scale, with measured outcomes where available; the role emphasizes operating systems with complex failure modes.
Practice an agent-runtime design exercise
As a preparation exercise, sketch lifecycle management, durable execution, checkpointing, scheduling, and the boundary between runtime state and durable memory. Explain how the runtime would connect to tools, data, other agents, and the platform’s gateway, IAM, and observability capabilities.
Rehearse a failure-and-load scenario
Trace a long-running, parallel workflow through a worker failure or overloaded dependency. Show how you would use backpressure, rate limits, idempotency, retries with jitter, circuit breakers, and dead-letter queues, and state the trade-offs behind your choices.
Prepare technology trade-off decisions
Draft a short comparison of a durable workflow engine, an agent framework, and any abstraction layer you would introduce. Be ready to discuss Temporal as the stated preference, framework portability, MCP or A2A where relevant, and LLM routing, structured outputs, streaming, and provider failover.
Bring a production-readiness artifact
Prepare a sample test plan, operational runbook, or architecture decision record drawn from your own work or clearly labeled as an exercise. Cover tracing, structured logs, health checks, deployment safety, and collaboration with SRE or platform teams, all emphasized in the posting.
Select leadership examples
Prepare specific examples, if you have them, of leading a system-design decision and improving code review, testing, API contracts, documentation, or mentoring. Explain the alternatives considered and how you communicated the decision.
Write questions for the platform team
Ask how the early-stage agent runtime is expected to interact with gateway, agentic IAM, memory, and observability; which reliability or latency constraints are most pressing; and where framework or protocol choices remain open. These are questions to clarify, not assumptions about GIC’s current implementation.

Availability last checked: 4 Nov 2026. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy

KEEP EXPLORING

Similar AI jobs

Related skills and specializations in Singapore. Matched to this role, not your profile.

Explore more jobs

Finding similar opportunities…