New opportunity

AI Solutions Engineer

Bob.pro · Raleigh, United States

About this role

Bob.pro is hiring a Raleigh-based Senior/Staff AI Solutions Engineer (Delivery Lead) to own client AI deployments from technical discovery and system design through coding, production, evaluation, and measurement. The engineer works alongside a CX and Product partner who owns the client relationship, while translating lessons from each deployment into reusable platform capabilities. The company builds governed AI operators for small and mid-market businesses.

Responsibilities

Spend roughly 50% of time writing production code, 25% in client-facing discovery, workflow mapping, design reviews and go-lives, 15% improving the platform, and 10% on design review, mentoring and delivery leadership. Typically work across two to three concurrent client accounts.
Own technical discovery, deployment, evaluations, guardrails, instrumentation and technical relationships with client operators and technical counterparts. Demonstrate what deployed systems did and what they cost.
Build around identity and state, tenant-scoped memory and context, dynamically constructed reasoning playbooks, and tenant-specific tools and actions. Favor configuration over client-specific code forks.
Maintain meaningful unit, integration and end-to-end tests; versioned schema migrations; verified tenant isolation; secure APIs and infrastructure; current documentation; and attributable, budgeted and reversible agent actions. The CX and Product partner, rather than this engineer, owns renewals, pricing, invoicing and routine relationship management.

Required qualifications

At least 8 years building software, including at least 3 years personally accountable for shipping and operating production systems. Recent hands-on production coding is essential.
Experience shipping LLM-powered systems used by real users, such as RAG or structured agents; building multi-tenant systems with isolation, data partitioning and per-tenant configuration; and incorporating AI evaluations, guardrails and rollback paths.
Ability to lead technical discovery with nontechnical business owners, write clearly, and connect system performance to a business outcome.

Preferred

Experience with OpenClaw, Claude Agent SDK, MCP or comparable agent runtimes; forward-deployed or consulting work; small or mid-market business domains such as field services, healthcare or professional services; and converting delivery patterns into platform features.

Benefits include Sedera health coverage paired with Direct Primary Care and unlimited PTO. Applicants send a resume and a one-page account of a production AI system they shipped to careers@bob.pro.

For full details about this role, click “Apply on company site” to view the employer’s original posting.

Skills for this role

AI agentsLLMsRAGTool callingMulti-tenant architectureData partitioningAI evaluationOffline evaluationShadow modeRegression testingGuardrailsRollbackAPI securityUnit testingIntegration testingEnd-to-end testingSchema migrationsEmbeddingsTechnical discoveryWorkflow mappingProduction deploymentInstrumentationTechnical writingMentoringOpenClawClaude Agent SDKMCP

Checking your profile…

Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.

Understand the role

What matters most

Own discovery through deployment
Source-backed: The engineer joins client discovery, maps workflows, designs the solution, writes production code, and owns go-live and technical follow-through. Prepare to explain how you turn a nontechnical operator’s problem into a design and then ship it yourself.
Prove outcomes, not just functionality
Source-backed: The role owns evaluations and instrumentation and must show clients what the system did, what it cost, and whether a business-relevant number moved. Prepare a concise account of how a shipped AI system was measured, including what did not work.
Build governed, operable AI systems
Source-backed: The production bar includes meaningful tests, versioned migrations, verified tenant isolation, secure APIs, current documentation, and attributable, budgeted, reversible agent actions. Be ready to explain concrete safeguards and rollback decisions from production work.
Turn deployments into reusable platform capability
Source-backed: About 15% of the role is platform work across identity and state, tenant-scoped memory, dynamic playbooks, and tenant-specific tools. The stated preference is configuration over client-specific code forks. Prepare an example of how a delivery pattern became reusable.
Lead technically while remaining hands-on
Source-backed: About half the role is coding and about 10% is design review, mentoring, and delivery leadership, alongside work on two to three accounts. The CX and Product partner owns the broader client relationship; this engineer owns technical discovery and delivery.

Likely challenges to think through

Balancing parallel deliveries with engineering rigor
Preparation hypothesis: Two to three concurrent accounts and a roughly 50% coding allocation may make context switching difficult without weakening tests, documentation, or release controls. Prepare to describe how you sequence work and protect the stated production bar under competing deadlines.
Customizing without creating code forks
Preparation hypothesis: Tenant-specific workflows and tools may pull against the stated configuration-first platform approach. Prepare a design example that distinguishes reusable primitives from tenant configuration, including where isolation boundaries are enforced and verified.
Explaining uncertain AI behavior in business terms
Preparation hypothesis: A client-facing engineer may need to reconcile imperfect model performance with an operator’s need for dependable results and clear costs. Prepare to walk through evaluation results, guardrail or rollback decisions, and how you would communicate outcomes without overstating them.
Sharing the client room without owning the relationship
Preparation hypothesis: The engineer must ask firsthand technical questions while the CX and Product partner runs the room and owns commercial matters. Prepare an example of surfacing a technical constraint or risk clearly, then handing off relationship or pricing decisions to the partner.

Practice your answers

Technical practice

Suggested practice question: How would you turn a business owner's workflow into a deployable AI system design?

What to coverThe role owns technical discovery and design alongside a CX and Product partner. Explain how you would map decisions, data, exceptions and success measures, then identify what needs validation before coding.

Suggested practice question: How would you enforce tenant isolation across an AI operator?

What to coverThe page calls for strict organization-level isolation across identity, state, partitioned data and tools. Discuss boundaries at each layer, how you would verify them, and the failure modes of per-tenant configuration.

Suggested practice question: How would you scope retrieval and memory to the right tenant and interaction?

What to coverThe stated design includes tenant-scoped transcripts, documents, embeddings and artifacts. Address ingestion and retrieval boundaries, context relevance, and tests that could catch cross-tenant exposure.

Suggested practice question: How would you construct and maintain tenant-specific reasoning playbooks?

What to coverThe page describes instructions built dynamically from each tenant's methodology, customer profile and vocabulary. Explain how configuration would shape those instructions and how you would test changes without creating client-specific code forks.

Suggested practice question: How would you make agent tool actions governable?

What to coverThe production bar requires every action to be attributable, budgeted and reversible, with tenant-specific tools loaded through an orchestrator. Cover authorization, action records, spending limits and recovery when an action produces an unwanted result.

Suggested practice question: How would you evaluate an LLM workflow before and after release?

What to coverThe page names offline evaluations, shadow mode, scoring and regression suites rather than informal testing. Describe what cases and measures you would use, how you would compare versions, and what evidence would block or support a rollout.

Suggested practice question: How would you deploy guardrails and respond when a production AI system fails?

What to coverThe role calls for guardrails and rollback paths that work under pressure. Explain how you would detect harmful behavior, limit its impact, decide when to roll back, and verify recovery.

Suggested practice question: What testing and migration plan would support a production deployment?

What to coverThe stated bar includes meaningful unit, integration and end-to-end tests plus versioned schema migrations instead of manual production database changes. Explain which risks each test level covers and how you would release a schema change safely.

Suggested practice question: How would you demonstrate what an operator did, what it cost and whether it helped?

What to coverThe engineer owns instrumentation and shows client owners system activity and cost while connecting performance to a business outcome. Describe the action and cost records you would collect, the outcome measure you would agree on, and how you would explain an inconclusive result.

Suggested practice question: When should a client deployment become a reusable platform capability?

What to coverThe role calls for feeding deployment lessons into identity, memory, playbooks and tool surfaces while favoring configuration over forks. Discuss how you would distinguish a repeatable pattern from a tenant-specific exception and weigh reuse against delivery risk.

Behavioural practice

Suggested practice question: Tell me about a time you clarified an ambiguous business workflow directly with its operator.

What to coverUse situation, action and result: explain what was unclear, the questions you asked, and how the answers changed the technical design. The page emphasizes firsthand discovery with nontechnical owners; avoid claiming an outcome you cannot substantiate.

Suggested practice question: Tell me about a production AI failure you had to contain.

What to coverUse situation, action and result: describe the signal you saw, the guardrail or rollback decision, and what you learned after recovery. This reflects the role's accountability for operating deployed systems under pressure.

Suggested practice question: Describe a time you worked with a client-facing partner while owning the technical delivery.

What to coverUse situation, action and result: show how you shared findings, handled a technical concern and kept responsibilities clear. The page assigns the client relationship to CX and Product while the engineer owns technical discovery and deployment.

Suggested practice question: Tell me about a time you raised the engineering standard while still shipping code.

What to coverUse situation, action and result: describe a design review or mentoring intervention, the change you made personally, and its effect on delivery quality. The role combines hands-on production coding with technical leadership.

Suggested practice question: Describe how you handled competing demands across concurrent deliveries.

What to coverUse situation, action and result: explain how you prioritized client needs, production risk and platform work, and what happened afterward. The page describes two to three concurrent accounts and time split between coding, client work, platform improvement and leadership.

Sharpen your knowledge

Topics to prepare

Tenant-scoped agent architecture
The posting names identity and state, partitioned tenant memory and context, dynamic playbooks, and tenant-specific tools as design primitives. Suggested exercise: sketch an interaction from identity check through context retrieval and action selection; mark every point where tenant isolation must hold.
Evaluation and regression strategy
The role requires built-in evaluation, including offline evals, shadow mode, scoring, and regression suites. Suggested exercise: outline how you would compare a proposed agent change with the deployed version, including failure cases and a decision rule for release.
Governed actions and rollback
The production bar requires every agent action to be attributable, budgeted, and reversible; guardrails and rollback paths are also required. Suggested exercise: trace an erroneous action from its audit record through containment, reversal, and prevention of recurrence.
Production delivery standards
The posting specifies meaningful unit, integration, and end-to-end tests; versioned schema migrations; verified isolation; secure APIs and infrastructure; and current documentation. Suggested exercise: draft a release checklist that explains what each control catches and how it is verified.
Discovery and business measurement
The engineer conducts technical discovery with nontechnical owners and must connect system performance to a business number. Suggested exercise: turn a vaguely described business workflow into questions, a proposed design, one measurable outcome, and a plan to explain results and costs to the owner.

Skills to revise

Implement a tenant-configured agent workflow
The role is roughly half production coding and favors configuration over client-specific forks. Suggested exercise: build a small structured-agent or tool-calling workflow with two tenant configurations; demonstrate that each tenant receives only its own context and permitted actions.
Verify isolation and migrations in code
The posting requires tenant isolation to be verified at every layer and schema changes to be versioned. Suggested exercise: add unit, integration, and end-to-end checks for cross-tenant access, then rehearse a versioned schema change without manually editing production data.
Build an evaluation harness
Offline evaluation, shadow mode, scoring, and regression testing are explicitly named. Suggested exercise: assemble representative workflow cases, score expected behavior, record regressions, and describe how shadow results would affect a go-live decision.
Instrument actions and rehearse recovery
This engineer owns instrumentation, guardrails, and deployment, and must show clients what a system did and cost. Suggested exercise: instrument a sample action with tenant, rationale, outcome, and cost records; trigger a guardrail and walk through rollback.
Run a technical discovery simulation
The posting says discovery with a nontechnical business owner is tested directly. Suggested exercise: practice a short interview that maps the current workflow, exceptions, permissions, and success measure, then explain the resulting design without relying on AI jargon.
Write a concise delivery readout
Clear writing, documentation, and turning deployment lessons into platform capability matter in this role. Suggested exercise: write a one-page readout covering what shipped, who used it, measured results, what broke, and which reusable configuration or platform change you would make next.

Your preparation checklist

Write the requested one-page account
Prepare a page about a production AI system you shipped: what it did, who used it, how you knew it worked, and what broke. The posting requests this alongside a resume and emphasizes the failure account.
Rehearse a technical discovery conversation
Practice asking a nontechnical business owner about their workflow, constraints, exceptions, and desired outcome. Finish with a short, structured design summary; the posting says this skill is tested directly.
Sketch a tenant-safe deployment
As a preparation exercise, diagram identity and state, partitioned tenant context, dynamic playbooks, and tenant-specific tools. Show where isolation is enforced and how configuration avoids client-specific code forks.
Prepare an evaluation and recovery walkthrough
Use a shipped system to explain offline or shadow evaluations, regression checks, guardrails, and a rollback decision. Be ready to describe what failed and what you changed, without implying every technique was used.
Show the production engineering evidence
Select concrete examples of recent code, meaningful unit/integration/end-to-end tests, versioned migrations, secure APIs, documentation, and attributable, budgeted, reversible actions where applicable.
Connect system behavior to a business number
Prepare a concise account of what a deployed system did, what it cost, and how a business-relevant measure changed—or did not. Explain the instrumentation and limits of your conclusion.
Practice a bounded work session
The posting describes a 90-minute session using an anonymized client problem followed by a repo review. Rehearse clarifying assumptions, building a small working slice, testing it, and explaining tradeoffs and unfinished work.
Bring role-specific questions
Ask how deployment lessons become platform features, how engineering and the CX/Product partner divide decisions during discovery, and how the team measures delivery across two to three concurrent accounts.

Availability last checked: Nov 4, 2026. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy

KEEP EXPLORING

Similar AI jobs

Related skills and specializations in United States. Matched to this role, not your profile.

Explore more jobs

Finding similar opportunities…