New opportunity

AI & Support Automation - Engineer

CertifyOS · Bengaluru, India

About this role

CertifyOS seeks a hands-on engineer to lead support automation and personally build, deploy and improve AI solutions for its healthcare provider-data platform. The position is remote, listed in Bengaluru, India, and full-time.

Responsibilities

Identify high-impact customer-support processes for automation; design, integrate, test and deploy AI-powered workflows and agents using LLMs, RAG and workflow automation.
Build end-to-end pipelines spanning knowledge and data ingestion, orchestration, model execution, tools and actions, evaluation and production monitoring. Integrate solutions with CRM systems, ticketing platforms, knowledge bases and internal tools.
Track quality, accuracy, cost, latency and business impact; continuously improve deployed systems and partner with Product and Engineering on AI-led transformation.

Experience and required skills:

The ideal experience is 3+ years overall in software engineering, AI/ML, automation, technical operations or support technology, including 2+ years of hands-on generative AI or LLM application development with production experience. Experience in automation, integrations or workflow engineering and taking projects from concept and architecture through production and improvement is sought.
Strong Python and/or JavaScript/TypeScript; LLM APIs, RAG, embeddings, vector databases, knowledge pipelines, AI agents, agentic workflows, tool or function calling and MCP. The role also requires REST APIs, webhooks, SQL, system integrations, LLM evaluation, observability, guardrails, monitoring, cost and latency optimization, CI/CD, production deployment, troubleshooting and understanding of customer-support workflows.
Candidates should be able to explain AI or automation solutions they personally built or led, their architecture and contribution, and measurable outcomes such as ticket deflection, average handling time reduction or cost savings.

Preferred tools include OpenAI, Anthropic, Claude, Gemini, Claude Code, LangChain, LangGraph, LlamaIndex, n8n, Make, Zapier, Power Automate, AWS, Azure, GCP, Docker, AI evaluation and observability tools, and CRM or helpdesk platforms including Zendesk, Salesforce, Jira, Freshdesk and Zoho. India-based employees receive health insurance, statutory leave and additional wellness leave for women; the company states it covers 100% of employee health, dental and vision insurance premiums.

For full details about this role, click “Apply on company site” to view the employer’s original posting.

Skills for this role

PythonJavaScriptTypeScriptGenerative AILLM APIsRAGEmbeddingsVector databasesKnowledge pipelinesAI agentsAgentic workflowsTool callingFunction callingModel Context Protocol (MCP)Workflow automationREST APIsWebhooksSQLSystem integrationLLM evaluationObservabilityAI guardrailsProduction monitoringCost optimizationLatency optimizationCI/CDProduction deploymentTroubleshootingCustomer-support workflowsOpenAI models and APIsе

Checking your profile…

Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.

Understand the role

What matters most

Choose support work worth automating
The role identifies high-impact support processes and is expected to quantify impact. Prepare to explain how you would assess a workflow using measures such as automation rate, ticket deflection, average handling time and cost savings.
Build AI solutions end to end
This is a hands-on role spanning knowledge ingestion, orchestration, LLMs or RAG, tool actions, evaluation and production deployment. Be ready to explain the architecture, your personal contribution and the production results of a solution you built or led.
Connect agents to support systems
The job calls for integrations with CRM, ticketing, knowledge bases and internal tools, using APIs, webhooks and other system-integration methods. Prepare a concrete account of how an automation moves from a support request to a reliable tool action.
Improve systems after launch
Monitoring quality, accuracy, cost, latency and business impact—and improving deployed systems—is an explicit responsibility. Prepare to describe evaluation, observability, troubleshooting and changes made in response to measured results.
Lead through hands-on collaboration
The engineer is expected to lead support automation initiatives while personally building solutions and partnering with Product and Engineering. Prepare to distinguish your own technical decisions from shared prioritization and delivery work.

Likely challenges to think through

Automating without losing answer accuracy
A likely trade-off is broader ticket deflection versus dependable answers from provider-data knowledge that may change. This is a preparation hypothesis, not a reported problem. Prepare to discuss retrieval evaluation, guardrails and when a workflow should hand off rather than answer.
Making cross-system actions dependable
Connecting agents to ticketing, CRM and internal tools may make automation useful while adding failure points and unintended-action risk. This is an inference from the integration duties. Prepare to explain tool permissions, validation, error handling and monitoring in an end-to-end design.
Balancing service quality with operating cost
The posting requires tracking accuracy, latency, cost and business impact; improving one measure may worsen another. Prepare an example of how you would set a baseline, evaluate a change and decide whether its support-workflow benefit justifies the added time or expense.

Practice your answers

Technical practice

Practice: How would you identify which customer-support process to automate first?

What to coverThe role includes identifying high-impact support processes for AI automation. Suggested preparation: compare ticket volume, handling time, repeatability and error risk, then explain how you would validate the expected benefit before building.

Practice: How would you design an end-to-end AI support workflow?

What to coverThe posting calls for pipelines spanning ingestion, orchestration, model execution, tools, evaluation and production monitoring, using Python and/or JavaScript/TypeScript. Suggested preparation: walk through the components and data flow, including where an LLM API is called and how failures are handled.

Practice: How would you build and maintain a RAG pipeline for support knowledge?

What to coverRAG, embeddings, vector databases and knowledge pipelines are stated skills. Suggested preparation: explain ingestion and updates, retrieval quality, and the trade-off between returning a grounded answer and declining when the retrieved material is insufficient.

Practice: When would you use an agent rather than a simpler workflow?

What to coverThe role names AI agents, agentic workflows, tool or function calling and MCP. Suggested preparation: compare flexible tool selection with predictable orchestration, and explain how you would limit and verify actions before they affect support systems.

Practice: How would you integrate an AI workflow with ticketing and internal systems?

What to coverThe posting specifies CRM, ticketing, knowledge-base and internal-tool integrations, along with REST APIs, webhooks and SQL. Suggested preparation: describe how you would move information between systems reliably, handle duplicate or failed events, and check that updates reached the intended record.

Practice: How would you evaluate answer quality and guard against unsafe automation?

What to coverLLM evaluation, accuracy and guardrails are explicit parts of the role. Suggested preparation: distinguish retrieval errors from generation errors, describe representative test cases, and explain when a workflow should avoid taking an action or route a case to a person.

Practice: How would you test and release a change to a production AI workflow?

What to coverThe role covers testing, CI/CD and production deployment. Suggested preparation: explain which pipeline, integration and AI-behaviour checks you would run before release, plus how you would detect a regression and roll back a problematic change.

Practice: How would you troubleshoot a deployed workflow whose answers have deteriorated?

What to coverProduction observability, monitoring and troubleshooting are required. Suggested preparation: trace ingestion, retrieval, model output and tool calls to isolate the failure, then explain how you would confirm that a fix improved results.

Practice: How would you reduce cost or latency without sacrificing support quality?

What to coverThe posting explicitly calls for monitoring and optimizing cost, latency and accuracy. Suggested preparation: discuss where time and spend arise across retrieval, model calls and tool actions, and how you would measure quality before and after a change.

Practice: How would you measure whether a support automation delivers business value?

What to coverThe posting cites automation percentage, ticket deflection, average handling time, cost savings and productivity gains as possible outcomes. Suggested preparation: choose metrics that fit the workflow, define a baseline, and explain how you would check for misleading gains that mask poorer support quality.

Behavioural practice

Practice: Tell me about an AI or automation solution you took from idea to production.

What to coverThe role emphasizes hands-on ownership from architecture through deployment and improvement. Suggested preparation: use a real situation, your own design and implementation actions, and the observed result; distinguish your contribution from the team's.

Practice: Describe a time you aligned Product and Engineering on an automation decision.

What to coverPartnering with Product and Engineering is a stated responsibility. Suggested preparation: give a real situation with differing priorities, the actions you took to reach a decision, and the resulting outcome.

Practice: Tell me about a production issue in an automation you worked on.

What to coverThe posting calls for production monitoring and troubleshooting. Suggested preparation: explain the situation and impact, your diagnostic and recovery actions, and the result, including what changed afterward.

Practice: Describe a time feedback or measured results changed a solution you built.

What to coverContinuous improvement, openness to feedback and measurable impact appear in the posting. Suggested preparation: describe a real situation, the feedback or evidence you acted on, what you changed, and the result without assuming the change was successful.

Sharpen your knowledge

Topics to prepare

Support automation opportunities
The posting calls for identifying high-impact support processes. As a preparation exercise, compare two hypothetical ticket types by volume, handling time, error risk and automation potential; explain which you would automate first and why.
RAG and knowledge quality
RAG, embeddings, vector databases and knowledge pipelines are must-have skills. Revise how ingestion, retrieval and source freshness affect answer accuracy; sketch how you would diagnose an answer grounded in outdated knowledge.
Agents, tool calling and MCP
The role explicitly includes agentic workflows, tool/function calling and MCP. Trace a hypothetical support request from model decision to tool action, and explain when to allow an action, request review or stop the workflow.
Evaluation and production trade-offs
The posting requires evaluation, guardrails, observability and monitoring of quality, accuracy, cost and latency. Define a small test set for a hypothetical support workflow and explain how you would investigate a quality drop after deployment.
End-to-end architecture
The role spans ingestion, orchestration, model execution, tools/actions, evaluation and production. Draw those stages for a hypothetical support automation flow and identify where failures and monitoring signals would appear.

Skills to revise

Build an LLM workflow in code
Strong Python and/or JavaScript/TypeScript and LLM API development are required. As practice, implement a small hypothetical workflow that accepts a support question, retrieves relevant knowledge and produces an answer; test empty retrieval and API errors.
Integrate support systems
The posting requires REST APIs, webhooks, SQL and integrations with CRM, ticketing, knowledge bases and internal tools. Practice mapping a hypothetical incoming ticket through a webhook, a SQL lookup and a ticket update, including error handling.
Test and measure an automation
The role calls for testing, evaluation and measurable impact. Create example support cases, record expected versus actual outcomes, and calculate a sample ticket-deflection or average-handling-time measure; label sample figures as hypothetical rather than personal results.
Deploy and troubleshoot
CI/CD, production deployment, monitoring and troubleshooting are must-have skills. Rehearse a deployment checklist for a hypothetical AI workflow, then diagnose a simulated latency spike using logs, model-call timing and integration responses.
Explain a personally built solution
The posting asks candidates to explain solutions they personally built or led, their architecture, contribution and quantified impact. Prepare a concise account using only real work you can substantiate; distinguish your decisions from team contributions and state how each outcome was measured.

Your preparation checklist

Map a support automation opportunity
Review CertifyOS’s stated provider licensing, enrollment, credentialing and network-monitoring products. Prepare a hypothetical support workflow, the manual steps it could reduce, and criteria for deciding whether automation is appropriate; do not present it as an existing company process.
Prepare a production project walkthrough
If you have a relevant AI or automation project, bring a one-page architecture showing the problem, your personal contribution, integrations, deployment and subsequent improvements. Include measured results such as ticket deflection, handling-time reduction or cost savings only where you have evidence.
Sketch an end-to-end agent design
Practice designing a hypothetical support workflow from knowledge ingestion and RAG through orchestration, model calls, tool actions and human handoff. Explain choices around embeddings, vector search, tool or function calling, and MCP where relevant.
Show how the workflow connects to other systems
Prepare a small diagram or code excerpt, if available, demonstrating Python or JavaScript/TypeScript with REST APIs, webhooks and SQL. Be ready to explain how you would test a connection to a ticketing system, CRM or knowledge base without assuming CertifyOS uses a particular vendor.
Define a production evaluation plan
Write a compact test-and-monitoring plan covering answer accuracy, guardrails, failure cases, observability, cost, latency and business impact. Include how CI/CD, troubleshooting and feedback would support improvements after deployment.
Bring questions that resolve design unknowns
Ask which support processes are highest priority, what knowledge sources and integrations are available, how success is measured, and how Product and Engineering collaborate on deployment and ongoing ownership.

Availability last checked: 5 Nov 2026. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy

KEEP EXPLORING

Similar AI jobs

Related skills and specializations in India. Matched to this role, not your profile.

Explore more jobs

Finding similar opportunities…