New opportunity

AI Solutions Engineer

College of the Holy Cross · Worcester, United States of America

About this role

The AI Solutions Engineer builds and supports AI-powered applications, agents, and integrations for teaching, learning, and College operations. This individual contributor works within architectural direction set by the Principal AI Solutions Engineer, delivering instrumented, supportable software in small increments and growing toward greater architectural ownership. The role is based in Worcester, MA, and is eligible for hybrid work.

Responsibilities

Build, test, and deploy agents that plan, call tools, and complete multi-step tasks, with human oversight where judgment or risk requires it. Implement APIs, middleware, and tool interfaces that enable safe actions on College systems. Design bounded workflows with clear permissions, retries, and fallbacks, using frameworks such as the Claude Agent SDK and Google ADK.
Create evaluation sets and test harnesses to measure behavior and detect regressions. Implement guardrails, permission scoping, and auditable logging; monitor production agents for cost, latency, errors, and behavioral drift. Maintain existing applications and agents.
Contribute to design and technical reviews; collaborate on faculty-facing work and the AI Exploration Lab. Coordinate with Enterprise Systems, Data Architecture, and Information Security. Gather staff and faculty requirements, estimate technical tasks, and document agent behavior, permissions, and operations.

Required qualifications

Bachelor’s degree in computer science, artificial intelligence, or a related field, or equivalent professional experience; at least two years of software or AI engineering experience, including hands-on development and deployment of large-language-model-powered applications.
Experience building agentic systems with tool and function calling, multi-step workflows, and orchestration across services; working proficiency with an agent framework such as the Claude Agent SDK or Google ADK.
Version control, automated testing, CI/CD, cloud deployment using a platform such as AWS, GCP, or Firebase, and integration with APIs and enterprise platforms. Ability to explain technical choices to non-engineers.

Preferred

Familiarity with AI evaluation and guardrail practices, and experience in higher education.

The salary range is $114,000–$121,000, commensurate with education, experience, and qualifications. The schedule is 40 hours per week, 12 months per year. An updated resume and cover letter are required for the internal application.

For full details about this role, click “Apply on company site” to view the employer’s original posting.

Skills for this role

AI agentsLarge language modelsTool callingFunction callingMulti-step workflowsService orchestrationClaude Agent SDKGoogle ADKAPI integrationMiddlewareAI evaluationTest harnessesAI guardrailsHuman-in-the-loop systemsLoggingVersion controlAutomated testingCI/CDAWSGCPFirebaseCloud deploymentEnterprise platform integrationInformation securityTechnical documentationRequirements gatheringTechnical communication

Checking your profile…

Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.

Understand the role

What matters most

Deliver agents that complete tasks
The posting emphasizes agents that plan, call tools, and complete multi-step work, rather than simply answer questions. Prepare to explain how a bounded workflow moves from a defined task through tool calls to completion or fallback.
Make actions on College systems safe
The role implements API and middleware interfaces, scoped permissions, human checkpoints, guardrails, and auditable logging. Prepare to explain where an agent may act, where a person must approve, and how failures are contained.
Measure behavior and support production agents
The posting calls for evaluation sets, regression harnesses, and monitoring of cost, latency, errors, and behavioral drift, alongside maintenance of existing agents. Prepare to describe how test results and production signals guide fixes.
Release supportable software in small increments
Within architectural direction set by the Principal AI Solutions Engineer, this individual contributor turns defined problems into instrumented software. The role also calls for automated testing, CI/CD, cloud deployment, and operational documentation.
Translate campus needs into coordinated delivery
The engineer gathers staff and faculty requirements, estimates technical tasks, contributes to reviews, and coordinates with Enterprise Systems, Data Architecture, and Information Security. Faculty-facing work and the AI Exploration Lab are also named.

Likely challenges to think through

Balancing autonomy with human control
Likely challenge, not a confirmed problem: more autonomous tool use can complete work faster but raises the stakes of an incorrect action. Prepare a concrete design rationale for permission boundaries, approval checkpoints, retries, and a fallback.
Moving quickly without losing reliability
Likely challenge, not a confirmed problem: small releases may compete with the time needed to evaluate agent behavior and detect regressions. Prepare to explain how you would scope an initial release, test it before deployment, and monitor it afterward.
Reconciling different stakeholders' needs
Likely challenge, not a confirmed problem: faculty and staff goals may need translation into work that fits architectural direction and enterprise security and data constraints. Prepare to explain how you would clarify requirements, surface trade-offs, estimate tasks, and document decisions for non-engineers.

Practice your answers

Technical practice

How would you design an agent to plan, call tools, and complete a multi-step task?

What to coverSuggested answer focus: Describe how the agent selects actions, passes results between steps, and determines when the task is complete. Explain how you would keep its scope bounded rather than treating it as a question-answering system.

How would you expose a College system safely to an agent?

What to coverSuggested answer focus: Discuss a tool or function interface built through an API or middleware, including input validation and limits on permitted actions. Explain how you would handle an unsuccessful call without assuming the agent may try a different action.

Where would you put human-in-the-loop checkpoints?

What to coverSuggested answer focus: Identify decisions or actions that warrant human judgment because of risk. Explain what information a reviewer would need and how the workflow would proceed after approval, rejection, or no response.

How would you make a multi-step agent workflow degrade gracefully?

What to coverSuggested answer focus: Cover bounded scope, sensible retries, and a defined fallback when a task cannot be completed. Distinguish a recoverable tool failure from a condition that should stop execution or seek help.

How would you evaluate an agent before releasing a change?

What to coverSuggested answer focus: Describe an evaluation set and test harness that exercise expected behavior and plausible failures. Explain how you would compare results across versions to catch regressions before users encounter them.

How would you make agent actions auditable and contain failures?

What to coverSuggested answer focus: Address guardrails, permission scoping, and logging of tool actions and outcomes. Explain the trade-off between useful operational records and limiting what an agent can access or expose.

How would you investigate a deployed agent whose behavior has changed?

What to coverSuggested answer focus: Start with the posting's production signals: cost, latency, error rates, and behavioral drift. Describe how you would use logs and evaluations to isolate a cause, tune the agent, and check that the change did not introduce a regression.

How would you use an agent framework for service orchestration?

What to coverSuggested answer focus: Use a framework you know, such as the Claude Agent SDK or Google ADK named in the posting, to explain how you would connect multi-step execution to tools and services. Distinguish what the framework handles from the permissions, tests, and failure behavior you must still implement.

How would you release an LLM-powered application in small increments?

What to coverSuggested answer focus: Describe a version-controlled path through automated tests, CI/CD, and cloud deployment, using a platform you know. Explain what you would verify in an initial limited release before expanding its capabilities.

What would you document to make a production agent supportable?

What to coverSuggested answer focus: Cover what the agent does, what it is permitted to touch, and how to operate it, as the posting specifies. Include how that documentation would help someone maintain an existing agent or respond when it cannot complete a task.

Behavioural practice

Tell me about turning stakeholder needs into a scoped technical task.

What to coverSuggested practice: Use a real situation involving staff, faculty, or another non-engineering stakeholder; explain how you clarified requirements and estimated the work. State what you delivered and what the stakeholder learned from the result.

Tell me about contributing to a design review within someone else's architectural direction.

What to coverSuggested practice: Describe the situation, your proposed technical choice, and how you responded to review or constraints set by a technical lead. Give the resulting decision and its effect on delivery or supportability.

Tell me about coordinating an integration across technical teams.

What to coverSuggested practice: Choose a real example involving teams with different responsibilities, such as systems, data, or security. Explain what you did to resolve an interface or risk question and what happened as a result.

Tell me about explaining a technical trade-off to non-engineers.

What to coverSuggested practice: Set out the audience's goal, the choices you needed to explain, and how you made the consequences understandable. Describe the decision or next step that followed without assuming agreement was automatic.

Tell me about delivering a small release while keeping software supportable.

What to coverSuggested practice: Describe a real situation where you narrowed scope, shipped an increment, and addressed testing, documentation, or operational needs. Explain what the release showed and how that informed the next increment.

Sharpen your knowledge

Topics to prepare

Agent planning and orchestration
Posting: Agents must plan, call tools, and complete multi-step work across services, rather than simply answer questions. Revision exercise (suggestion): Sketch a task flow showing tool selection, intermediate results, completion criteria, and where a human checkpoint belongs.
Safe, bounded agent actions
Posting: Workflows need clear permissions, sensible retries, defined fallbacks, and human oversight where judgment or risk requires it. Revision exercise (suggestion): For a hypothetical College-system action, specify the permitted operations, approval point, retry limit, and failure response.
Evaluation and guardrail principles
Posting: The engineer builds evaluation sets and test harnesses to catch regressions and implements guardrails and permission scoping; familiarity with these practices is also listed as preferred. Revision exercise (suggestion): Define test cases for correct tool use, unauthorized actions, and incomplete tasks, then state what each case should measure.
Production reliability and auditability
Posting: Deployed agents are monitored for cost, latency, errors, and behavioral drift; actions must be auditable, and existing agents need support. Revision exercise (suggestion): Explain which logs and measurements would help diagnose a failed multi-step run without expanding an agent’s permissions.
Incremental delivery and collaboration
Posting: The engineer follows architectural direction from the Principal AI Solutions Engineer, releases in small increments, and works with faculty, staff, Enterprise Systems, Data Architecture, and Information Security. Revision exercise (suggestion): Break a hypothetical faculty-facing request into a first release, review questions, technical tasks, and an estimate; explain the trade-offs in nontechnical language.

Skills to revise

Build a framework-based agent
Posting: Working proficiency with a framework such as the Claude Agent SDK or Google ADK is required. Practice exercise (suggestion): Use one named framework to implement a small agent that selects a tool, completes two steps, and pauses for human approval before a consequential action.
Implement a scoped tool interface
Posting: The role implements tool and function interfaces using APIs and middleware and integrates with enterprise platforms. Practice exercise (suggestion): Create a mock API-backed tool with input validation and restricted actions; demonstrate how an unauthorized call is rejected.
Write agent regression tests
Posting: The engineer builds evaluation sets and test harnesses and uses automated testing. Practice exercise (suggestion): Add tests for successful completion, incorrect tool selection, a tool failure, and fallback behavior; record expected versus observed results.
Exercise deployment and monitoring
Posting: Version control, CI/CD, and cloud deployment on a platform such as AWS, GCP, or Firebase are required; production monitoring covers cost, latency, and errors. Practice exercise (suggestion): Take a small agent through a version-controlled test-and-deploy workflow on one named platform, then inspect logs and those three measurements.
Document behavior and operating limits
Posting: The engineer gathers stakeholder requirements, estimates tasks, and documents what each agent does, what it may touch, and how to operate it. Practice exercise (suggestion): Write a one-page handoff for the practice agent covering its task, permissions, approval point, fallback, operating steps, and a plain-language explanation of a design choice.

Your preparation checklist

Prepare an agent project walkthrough
Choose a real LLM application you built and deployed. Map its tool calls, multi-step orchestration, framework choice, your contribution, and what happened in production; do not imply you used the Claude Agent SDK or Google ADK unless you did.
Sketch a bounded workflow design
For a hypothetical teaching, learning, or College operations task, draw an agent-to-API flow with scoped permissions, a human approval checkpoint, retries, and a fallback. This is a preparation exercise, not a stated interview assignment.
Show how you would measure agent behavior
Prepare a small example evaluation set and test-harness outline covering successful tasks, unsafe actions, and regressions. Explain what you would log and how you would monitor cost, latency, errors, and behavioral drift.
Collect evidence of reliable delivery
Identify examples of automated tests, version control, CI/CD, cloud deployment, and API or enterprise-platform integration. Be ready to explain a small release, a production issue, and how you supported the software afterward.
Practice turning stakeholder needs into a plan
Draft a brief example of clarifying a faculty or staff request, splitting it into estimated increments, and documenting agent behavior and permissions. Practice explaining the technical choices without engineering jargon.
Prepare application materials for the stated process
The page calls for an updated resume and cover letter as part of its internal application; it asks the letter to reflect on alignment with Holy Cross’s mission. Use specific, truthful examples rather than a generic values statement.
Write questions about delivery and ownership
Ask how the Principal AI Solutions Engineer’s architectural direction is translated into work, how faculty-facing ideas reach production, and how Enterprise Systems, Data Architecture, and Information Security participate in reviews.

Availability last checked: Nov 4, 2026. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy

KEEP EXPLORING

Similar AI jobs

Related skills and specializations in United States of America. Matched to this role, not your profile.

Explore more jobs

Finding similar opportunities…