New opportunity

AI Observability Engineer

AIG PC Global Services, Inc. · Atlanta, United States of America

About this role

AIG PC Global Services, Inc. seeks an AI Observability Engineer for its Generative AI team in Atlanta. The role works with AI product teams to monitor AI systems, support AI governance, and ensure solutions align with enterprise Responsible AI principles and standards.

Responsibilities

Track performance of machine learning, LLM, and agentic AI systems using measures such as context relevance, accuracy, precision, recall, and F1 score. Monitor input data for anomalies, bias, and drift; use observability tools to turn AI-system activity data into trustworthy insights.
Operate the AI model management platform and pipelines, and improve performance indicators and observability capabilities. Independently review AI solutions against organizational policies, the AI Governance Framework, and Responsible AI principles.
Support product teams through reviews, queries, and workshops. Coordinate Responsible AI reviews with legal, compliance, security, and other teams; prepare executive reports and strategic recommendations.
Create training for technical and nontechnical audiences, improve Responsible AI methods, tutorials, and tools, and translate emerging global AI regulations, standards, and policies into framework updates.

Required qualifications

At least 3 years of experience and a bachelor’s degree. Knowledge of machine learning and LLM models, algorithms, and evaluation metrics; data-pipeline and analysis skills, including cleaning and transforming data; familiarity with LLM development; and experience or interest in Responsible AI or AI ethics and Generative AI. The role also calls for cross-functional collaboration, knowledge of policies for emerging AI tools, complex problem-solving and mitigation strategies, ownership, and comfort with ambiguity. Experience in financial services or another regulated industry is ideal.

Preferred

Experience building production LLM, RAG, or model-inference monitoring pipelines; OpenTelemetry tracing or structured logs; Fiddler, Arize, LangSmith, or similar tools; and dashboards and alerts focused on model behavior.

Team members are asked to work primarily in the office. AIG describes benefits covering health, wellbeing, financial security, and professional development.

For full details about this role, click “Apply on company site” to view the employer’s original posting.

Skills for this role

Responsible AIAI governanceAI ethicsMachine learningLarge language modelsGenerative AIAgentic AIModel performance monitoringModel evaluationData quality monitoringBias detectionData drift detectionData pipelinesData analysisData cleaningData transformationLLM developmentRAGModel inference monitoringOpenTelemetryDistributed tracingStructured loggingFiddlerArizeLangSmithDashboardsAlertingAI regulatory frameworksCross-functional collaborationExecutive reporting

Checking your profile…

Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.

Understand the role

What matters most

Monitor AI model behavior
The role tracks effectiveness across ML, LLM, and agentic AI systems using measures including context relevance, accuracy, precision, recall, and F1.
Turn data signals into trustworthy insights
AIG calls for monitoring inputs for anomalies, bias, and drift, and using observability tools and data pipelines to analyze AI-system activity.
Operate and improve observability
The engineer manages the AI model management platform and pipelines while evolving performance indicators and observability capabilities. Production LLM or RAG monitoring, tracing, and behavior-focused alerts are listed as preferred experience.
Provide independent Responsible AI review
The role reviews solutions against organizational policies, the AI Governance Framework, and Responsible AI principles, coordinating issues with product, legal, compliance, and security teams.
Make governance actionable
Duties include executive reports and recommendations, training for technical and nontechnical audiences, improvements to Responsible AI methods and tools, and translating emerging standards and policies into framework updates.

Likely challenges to think through

Choosing useful signals across AI systems
Preparation hypothesis: a single metric may not describe ML classifiers, LLM responses, and agentic behavior equally well. Prepare to explain how you would select measures for each use case, validate what they reveal, and avoid treating a dashboard score as proof of safety or effectiveness.
Separating meaningful change from monitoring noise
Preparation hypothesis: anomaly, bias, and drift monitoring may require balancing missed issues against excessive alerts. Prepare a concrete approach to data-quality checks, baselines, investigation, and when to escalate for human review.
Balancing independent review with product delivery
Preparation hypothesis: reviewing solutions independently while supporting product teams and coordinating legal, compliance, and security perspectives can create competing priorities. Prepare to describe how you would document evidence, explain a mitigation clearly, and bring unresolved trade-offs to decision-makers without assuming a specific AIG approval process.

Practice your answers

Technical practice

How would you choose between accuracy, precision, recall, and F1 for monitoring an ML model?

What to coverPreparation suggestion: Explain what each metric captures and how the costs of false positives and false negatives affect the choice. Discuss how you would check whether an aggregate score hides poor performance on important subsets.

How would you evaluate context relevance and answer quality for an LLM?

What to coverPreparation suggestion: Distinguish whether retrieved context is relevant from whether the answer uses it accurately. Explain how you would assemble evaluation examples and use human review when an automated score is uncertain.

What would you monitor in an agentic AI system beyond its final response?

What to coverPreparation suggestion: Consider the sequence of actions and intermediate outputs needed to understand a failure, while noting the limits of automated evaluation. Explain when a concerning pattern should be escalated for human oversight.

How would you investigate anomalies, drift, and potential bias in model inputs?

What to coverPreparation suggestion: Describe baseline comparisons, data-quality checks, and relevant subgroup analysis. Explain how you would distinguish a data-pipeline fault from a genuine change in inputs before recommending mitigation.

How would you prepare pipeline data for trustworthy AI monitoring?

What to coverPreparation suggestion: Cover validation, cleaning, transformations, and consistent definitions for monitoring measures. Discuss how you would detect missing or inconsistent records so a dashboard does not present misleading conclusions.

How would you instrument an LLM or RAG monitoring pipeline to investigate failures?

What to coverPreparation suggestion: Discuss useful structured logs or traces across retrieval, inference, and response steps, and how you would connect them without assuming a particular platform. OpenTelemetry and AI observability tools are listed as nice-to-have experience, so treat them as possible approaches rather than established AIG tooling.

What model-behavior dashboards and alerts would you design beyond uptime?

What to coverPreparation suggestion: Choose a small set of quality, drift, and data-health signals linked to actionable thresholds. Explain the trade-off between detecting deterioration promptly and creating noisy alerts.

How would you improve KPIs while operating a model-management platform and its pipelines?

What to coverPreparation suggestion: Explain how you would assess whether existing measures reflect model effectiveness and remain reliable as systems change. Address operational checks and how you would evaluate a proposed KPI change before adopting it.

How would you conduct an independent Responsible AI review of an AI solution?

What to coverPreparation suggestion: Outline how you would gather evidence, compare it with applicable organizational policies and governance principles, and document gaps and mitigations. Avoid assuming the content of AIG's internal framework, which the posting does not specify.

How would you turn an emerging AI regulation or standard into a governance-framework update?

What to coverPreparation suggestion: Describe how you would verify applicability, identify affected practices, and work with legal, compliance, and security on an actionable change. Explain how you would communicate uncertainty rather than treating an emerging requirement as settled.

Behavioural practice

Tell me about a time you raised a concern during an independent review despite disagreement.

What to coverPreparation suggestion: Use a real situation to explain the evidence you gathered, the action you took to discuss a mitigation, and the result. Clarify how you maintained an independent assessment while working constructively with the product team.

Describe a time you coordinated people with different priorities on an AI risk or governance issue.

What to coverPreparation suggestion: Set out the situation, your actions to clarify ownership and reconcile concerns, and the result. Focus on relevant collaboration with functions such as legal, compliance, security, or product without claiming experience you do not have.

Give an example of taking ownership of an ambiguous monitoring or AI governance problem.

What to coverPreparation suggestion: Explain the situation, how you scoped the unknowns and chose an initial course of action, and the result. Include how you adjusted when new evidence emerged.

Tell me about a time you translated a technical risk into a recommendation for senior stakeholders.

What to coverPreparation suggestion: Describe the situation, the action you took to separate evidence from uncertainty and present options, and the result. Show how you made the recommendation useful without overstating what the data established.

Describe a time you adapted technical or Responsible AI guidance for different audiences.

What to coverPreparation suggestion: Use a real training, tutorial, or workshop situation, explain how you changed your approach for the audience, and give the result. Highlight how you checked whether participants could apply the guidance.

Sharpen your knowledge

Topics to prepare

Model evaluation measures
Revise when accuracy, precision, recall, F1 and context relevance are informative, and where each can mislead. Exercise: choose measures for an ML classifier and an LLM response, then explain what a change in each measure would mean. The posting explicitly calls for monitoring these KPIs.
LLM and agentic-system behavior
Revise how LLM outputs and agentic steps can be evaluated, including the limits of automated evaluation and the need for human oversight. Exercise: sketch what evidence a reviewer would need to assess a multi-step AI response. The role monitors LLM and agentic AI systems and highlights scaling human oversight.
Input quality, bias and drift
Revise the difference between anomalous inputs, distribution drift and potential bias, including their effects on fairness and accuracy. Exercise: given two periods of sample input data, identify signals worth investigating and explain what conclusions cannot yet be drawn. These are explicit monitoring responsibilities.
Responsible AI governance
Revise how an independent review can test an AI solution against organizational policies, a governance framework and Responsible AI principles. Exercise: draft review questions, evidence to request and possible mitigations for a hypothetical AI use case. The posting requires such reviews but does not disclose AIG’s internal criteria.
AI policy and regulatory change
Revise how to turn an emerging AI standard or policy change into a practical framework update. Exercise: summarize a hypothetical change, affected review questions and stakeholders to consult. The role explicitly tracks global developments and helps translate them into Responsible AI framework updates.

Skills to revise

Prepare monitoring data
Practice cleaning and transforming a small set of AI activity records, then define checks for missing, inconsistent and anomalous inputs. This directly reflects the required data-pipeline and analysis skills and the role’s ongoing pipeline operations.
Investigate a model-behavior signal
Practice tracing a drop in F1 or context relevance from the reported KPI back to input quality, evaluation data and possible drift; record uncertainties and a mitigation proposal. This mirrors the role’s monitoring and complex problem-solving responsibilities.
Conduct a cross-functional review
Practice writing a short independent review that states the observed Responsible AI issue, supporting evidence, open questions and decisions needed from legal, compliance and security. The posting explicitly calls for coordinating reviews with those teams.
Communicate findings to different audiences
Practice turning one monitoring finding into a brief executive report with a recommendation and a separate plain-language training explanation. The role produces executive reports and Responsible AI training for technical and nontechnical audiences.
Instrument behavior-focused monitoring
Optional preparation for the posting’s nice-to-have skills: sketch a production LLM or RAG monitoring pipeline using OpenTelemetry traces or structured logs, then specify a dashboard and alert for model behavior rather than uptime. These tools and production-pipeline experience are preferred, not stated as required.

Your preparation checklist

Map the role to evidence
Make a one-page evidence map for the stated requirements: ML and LLM evaluation, data pipelines and transformation, Responsible AI, cross-functional work, and mitigation strategies. Note where an example is available and where preparation is still needed.
Build a project walkthrough
Prepare a concise walkthrough of a relevant AI monitoring or governance project, if available: the system, your decisions, evidence collected, issues found, and outcomes. If no such project is available, label a proposed approach as hypothetical.
Work through an evaluation example
Prepare a small example showing how you would assess context relevance and appropriate accuracy, precision, recall, or F1 measures, then check inputs for anomalies, bias, and drift. Explain what each result can and cannot establish.
Sketch an observability design
Draw a proposed monitoring pipeline from cleaned inputs and model activity to metrics, dashboards, alerts, and human review. Discuss how structured logs or OpenTelemetry traces could help; the posting lists these as preferred experience, not required tools.
Practice an independent Responsible AI review
Draft a hypothetical review note identifying a risk, the evidence needed, a mitigation, and an unresolved decision. Show how you would coordinate with product, legal, compliance, and security teams without assuming AIG’s internal review procedure.
Prepare two communication samples
Create a short executive summary with a recommendation and a plain-language training outline for a technical or nontechnical audience. The role calls for both executive reporting and Responsible AI training.
Bring targeted employer questions
Write questions about the AI systems and model-management platform in scope, how monitoring findings feed governance reviews, who owns mitigation decisions, and how the team prioritizes framework updates and primarily in-office collaboration.

Availability last checked: Nov 4, 2026. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy

KEEP EXPLORING

Similar AI jobs

Related skills and specializations in United States of America. Matched to this role, not your profile.

Explore more jobs

Finding similar opportunities…