New opportunity

Staff Applied AI Engineer, Agents

Morena · Boston or San Francisco Bay Area, United States

About this role

Morena is seeking a staff-level, hands-on engineer to design, build, deploy, and operate production AI agents for complex enterprise workflows in a large regulated industry. The agents span communication, operational, and transaction-heavy processes and must handle multi-step reasoning, tools and APIs, business rules, state, and production reliability. This is an engineering and delivery role, not a traditional solutions engineering position.

Responsibilities

Own complex customer deployments from workflow discovery and agent architecture through integration, evaluation, debugging, and reliable production operation. Design agent orchestration, prompts and context, retrieval, structured workflows, external API integrations, state management, human escalation, and failure recovery. Decide when to use probabilistic AI versus deterministic application logic.
Measure and improve agent quality through evaluation datasets, automated checks, model-based evaluation, regression tests, monitoring, trace analysis, failure classification, and real-world outcome metrics. Investigate root causes across models, context, tools, integrations, infrastructure, and customer systems rather than relying only on prompt changes.
Turn deployment lessons into reusable components, templates, tooling, integration patterns, evaluation methods, and playbooks. Partner with Product and Platform engineering to distinguish customer-specific needs from reusable capabilities and product gaps, and inform the technical roadmap. Provide technical reviews, architecture guidance, and mentorship while continuing to write production code.

Requirements

At least 6 years of professional software engineering experience, including building and operating production software. Be comfortable with Python, APIs and services, cloud infrastructure, databases, distributed systems, integrations, observability, testing, and production operations.
Hands-on experience building modern LLM systems, including agentic systems, tool calling, prompt and context engineering, LLM workflows, retrieval, structured generation, model APIs, and agent frameworks. Production experience matters more than experimentation alone.
Strong engineering judgment about reliability, failure modes, evaluation, deployment, and operational risk. Work autonomously through difficult problems and communicate technical trade-offs directly with sophisticated customers and stakeholders; the primary responsibility remains shipping systems.

Additional relevant experience includes production agentic systems, AI evaluation or tracing, voice AI or conversational systems, workflow automation, B2B SaaS, regulated industries, third-party integrations, high-volume customer-facing systems, and turning bespoke work into platform capabilities.

This is a full-time hybrid role based in Boston or the San Francisco Bay Area, with approximately two office days per week and the remaining time available remotely. Candidates should be based in or able to work from either office. The compensation package includes salary, equity, and benefits; details are discussed during screening.

For full details about this role, click “Apply on company site” to view the employer’s original posting.

Skills for this role

PythonAI agentsLLMsAgent orchestrationTool callingPrompt engineeringContext engineeringRetrievalStructured generationModel APIsAgent frameworksAPI integrationState managementWorkflow automationAI evaluationRegression testingProduction monitoringTrace analysisFailure analysisObservabilityCloud infrastructureDatabasesDistributed systemsProduction debuggingTechnical leadershipCustomer-facing engineering

Checking your profile…

Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.

Understand the role

What matters most

Own deployments through production
The role owns complex customer deployments from workflow discovery and architecture through integration, edge-case resolution, and reliable production performance. Success is framed around outcomes, not a single implementation task.
Engineer reliable agent workflows
Agents must handle multi-step processes, tools and APIs, business rules, state, retrieval, human escalation, and failure recovery. A central design decision is where probabilistic AI belongs versus deterministic application logic.
Measure quality and diagnose failures
The role establishes evaluation datasets, automated and regression checks, monitoring, trace analysis, failure classification, and real-world outcome metrics. Debugging spans models, context, tools, integrations, infrastructure, and customer systems.
Turn deployments into reusable capabilities
Lessons from individual implementations should become shared components, templates, tooling, integration patterns, evaluation methods, and playbooks. The engineer works with Product and Platform teams to separate customer-specific needs from product gaps and reusable capabilities.
Lead technically while shipping code
This staff-level IC provides architecture guidance, technical reviews, and mentorship while continuing to own significant production engineering work and communicate trade-offs with sophisticated customers.

Likely challenges to think through

Balancing agent flexibility with controlled execution
Preparation hypothesis, not a confirmed employer problem: transaction-heavy workflows in a regulated industry may make flexible reasoning difficult to reconcile with business rules and recoverable failures. Prepare a design example showing how you would bound agent decisions, use deterministic logic, manage state, and escalate to a human.
Finding causes behind apparent model failures
Preparation hypothesis, not a confirmed employer problem: an incorrect outcome may originate in context, a failed tool call, an integration, or the evaluation method rather than the model. Prepare a production debugging example that traces evidence from observed outcome to root cause and verifies the fix with regression checks.
Scaling learning without overgeneralizing
Preparation hypothesis, not a confirmed employer problem: customer-specific workflows may resist standardization, while bespoke fixes slow future deployments. Prepare to explain how you would decide what becomes a shared pattern or platform capability and what should remain deployment-specific.

Practice your answers

Technical practice

Suggested practice: How would you design an agent for a multi-step enterprise workflow?

What to coverThe role involves agents that use tools, follow business rules, and operate in production. Describe how you would map the workflow, define orchestration and system boundaries, and test the design against realistic edge cases.

Suggested practice: Where should an agent use AI rather than deterministic logic?

What to coverThe posting explicitly calls for judgment about this boundary. Explain how you would weigh flexibility against predictability, especially for business rules and transaction-heavy steps.

Suggested practice: How would you make external tool and API calls reliable?

What to coverCustomer-system integrations and failed tool calls are part of the stated work. Discuss request validation, error handling, retries, and how you would distinguish an integration failure from an agent decision error.

Suggested practice: How would you manage state and recover a partially completed workflow?

What to coverThe agents must manage state and recover from failures across multi-step processes. Describe what state you would persist, how you would avoid repeating consequential actions, and when you would stop or escalate.

Suggested practice: How would you design context and retrieval for an agent?

What to coverPrompt and context design and retrieval are named responsibilities. Explain how you would select relevant information, handle missing or stale context, and check whether a context change improves behavior.

Suggested practice: How would you evaluate agent quality before and after deployment?

What to coverThe role calls for evaluation datasets, automated and model-based checks, regression tests, and real-world outcome metrics. Discuss what each can reveal, where each can mislead, and how you would decide whether a change is an improvement.

Suggested practice: How would you investigate an agent failure without defaulting to a prompt change?

What to coverThe posting emphasizes trace analysis and root-cause investigation across models, context, tools, integrations, and infrastructure. Walk through how you would classify the failure, gather evidence, and verify a fix.

Suggested practice: How would you operate an agent service reliably in production?

What to coverPython services, cloud infrastructure, databases, distributed systems, observability, and production operations are all cited. Explain what you would monitor, how you would diagnose a distributed failure, and how testing would support a safe deployment.

Suggested practice: When should an agent escalate to a human?

What to coverHuman escalation, business logic, guardrails, and a regulated-industry setting appear in the role description. Describe how you would identify an unsafe or unresolved step, preserve useful context for review, and measure whether escalation works as intended.

Suggested practice: How would you turn one deployment's solution into a reusable capability?

What to coverThe role includes creating shared components, integration patterns, evaluation methods, and playbooks. Explain how you would separate a customer-specific constraint from a general pattern and assess the cost of making that pattern reusable.

Behavioural practice

Suggested practice: Tell me about a time you clarified an ambiguous customer workflow.

What to coverUse a real situation, the questions you asked and actions you took, and the resulting engineering decision. The role requires translating operational problems into systems while working with technically sophisticated customers.

Suggested practice: Tell me about a production failure you owned through resolution.

What to coverSet out the situation, your diagnostic and recovery actions, and the operational result. Focus on how evidence changed your understanding of the failure and what you did to reduce recurrence.

Suggested practice: Tell me about a technical trade-off you communicated under pressure.

What to coverDescribe the situation, the options and risks you presented, your action, and the result. The posting emphasizes autonomous decisions and clear communication with customers and stakeholders.

Suggested practice: Tell me about a time you brought deployment lessons back to a product or platform team.

What to coverGive a real example of the situation, how you distinguished a one-off need from a reusable gap, and what resulted. Keep the focus on your actions and the evidence behind the recommendation.

Suggested practice: Tell me about a time you improved another engineer's design while still owning delivery.

What to coverDescribe the situation, your review or mentoring actions, and the outcome for the system and team. This reflects the role's combination of technical guidance and hands-on production work.

Sharpen your knowledge

Topics to prepare

Agent architecture and control boundaries
Posting: Agents must handle multi-step workflows, tool use, state, and business rules. Suggested revision: Diagram a hypothetical workflow and mark where an LLM makes a judgment versus where deterministic application logic controls the next step.
Context, retrieval, and structured outputs
Posting: The role includes prompt and context design, retrieval, and structured generation. Suggested revision: For a sample workflow, specify what information the agent needs at each step, how it is retrieved, and what output must be validated.
Production reliability and failure recovery
Posting: Deployments must operate reliably across APIs, databases, distributed systems, and customer integrations. Suggested revision: Trace a failed multi-step transaction and identify how state, retries, human escalation, and recovery should work.
Evidence-based agent evaluation
Posting: Quality work includes evaluation datasets, automated checks, model-based evaluation, regression tests, and outcome metrics. Suggested revision: Design a small test set with success criteria and explain how you would decide whether a change improved production behavior.
Reusable patterns and technical ownership
Posting: This staff-level IC turns deployment lessons into shared components and works with Product and Platform engineering. Suggested revision: Take a hypothetical customer-specific integration issue and explain which parts belong in that deployment, which could become reusable, and what trade-offs you would communicate.

Skills to revise

Implement a Python agent workflow
Posting: Hands-on Python, agent orchestration, tool calling, APIs, and production code are central. Suggested exercise: Build a small multi-step workflow with a tool call, structured response validation, explicit state transitions, and a deterministic business-rule check.
Test retrieval and context construction
Posting: Retrieval and context engineering are named requirements. Suggested exercise: Create a small set of sample records, assemble context for a question, and test how the workflow behaves when relevant information is missing or conflicting.
Build an agent regression suite
Posting: The role calls for evaluation datasets, automated checks, and regression tests. Suggested exercise: Write representative success and failure cases, run them against two workflow versions, and summarize behavior changes against predefined criteria.
Debug from traces to root cause
Posting: Production debugging spans models, context, failed tool calls, APIs, infrastructure, and customer systems. Suggested exercise: Construct a sample failed execution trace, classify the failure, identify evidence for its likely cause, and propose a fix plus a verifying test.
Review an integration for production operation
Posting: The engineer owns integrations, observability, testing, and reliable operation. Suggested exercise: Review a sample API integration for failure handling, state consistency, monitoring signals, and human escalation; explain the highest-risk trade-offs as you would in a technical review.

Your preparation checklist

Prepare a production deployment walkthrough
The role owns agent deployments from design through operation. Prepare a concise account of a system you personally built, if applicable, covering the workflow, your code and decisions, integrations, launch, and production outcomes; distinguish shipped work from prototypes.
Sketch an enterprise agent design
As a preparation exercise, diagram a hypothetical multi-step workflow with tool calls, retrieval, state, business rules, human escalation, and failure recovery. Mark where deterministic logic should replace probabilistic AI and explain why.
Build an evaluation plan
Prepare a sample quality plan with test cases, automated and model-based checks, regression tests, trace review, failure categories, and operational outcome metrics. Explain what evidence would justify a change.
Rehearse a production incident analysis
Use a real example you can discuss, or a clearly labeled hypothetical, to trace an incorrect agent action across context, model behavior, tool calls, APIs, and infrastructure. Show how you would isolate the cause and verify the fix.
Show how deployment lessons become reusable
Prepare one example or proposed design for turning a customer-specific integration or failure pattern into a shared component, evaluation method, or playbook. Explain the boundary between a bespoke need and a product or platform gap.
Prepare evidence of hands-on technical leadership
Select concrete examples, where available, of production Python and service work, technical reviews, architecture decisions, and mentorship. Be ready to explain trade-offs while keeping your own implementation contribution clear.
Draft customer and employer questions
Prepare questions about workflow discovery, integration constraints, quality measures, escalation paths, and how Product and Platform teams use deployment findings. The posting says customer interaction is required, but shipping production systems remains the primary responsibility.

Availability last checked: Nov 4, 2026. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy

KEEP EXPLORING

Similar AI jobs

Related skills and specializations in United States. Matched to this role, not your profile.

Explore more jobs

Finding similar opportunities…