New opportunity

Research Engineer I (AI Evaluation, Trust Technologies)

Nanyang Technological University · Singapore

About this role

The National Centre for Research in Digital Trust (DTC) at Nanyang Technological University (NTU) is seeking a Research Engineer to support the evaluation and validation of AI systems. This role is dedicated to ensuring that emerging technologies are trustworthy, fair, and safe in practice by assessing their performance in real-world conditions. Key Responsibilities: You will design evaluation frameworks, test datasets, and scoring methodologies for AI and ML systems. The role involves conducting error analysis, translating findings into actionable recommendations, and running structured experiments with large language models. You will also develop annotation guidelines, monitor labelling quality, and build supporting prototypes and tooling. Additionally, you will engage directly with project partners and clients, participating in technical meetings and communicating results to both technical and non-technical audiences. Job Requirements: Candidates must hold a Bachelor’s degree in computer science, engineering, data science, or a related field. Required technical skills include proficiency in Python and practical experience in ML workflows, specifically model evaluation and error analysis. You must possess a working knowledge of NLP and hands-on experience with large language models. Strong communication skills are essential for engaging with external partners. Preferred Qualifications: Experience in evaluation methodology or data annotation is considered a strong advantage. Candidates should be careful, methodical, and capable of working independently under tight timelines.

For full details about this role, click “Apply on company site” to view the employer’s original posting.

Skills for this role

PythonMachine LearningModel EvaluationError AnalysisNatural Language ProcessingLarge Language ModelsData AnnotationEvaluation Frameworks

Checking your profile…

Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.

Understand the role

What matters most

AI Evaluation Framework Design
The core of this role is building robust scoring methodologies and test datasets to validate AI systems. You must demonstrate a methodical approach to defining what 'trustworthy' and 'safe' mean in measurable, technical terms for diverse AI applications.
Error Analysis and Actionable Insights
Beyond identifying model failures, you are expected to translate complex error analysis into clear, actionable recommendations. This requires a deep understanding of ML workflows to pinpoint why models fail in real-world conditions.
Stakeholder Communication
You will act as a bridge between technical research and external partners. Success depends on your ability to explain complex AI evaluation results to non-technical audiences while maintaining professional engagement in project meetings.
Data Annotation and Quality Control
You are responsible for developing annotation guidelines and monitoring the quality of labelled data. This is critical for ensuring the integrity of the datasets used to train and evaluate the AI systems.

Likely challenges to think through

Translating Technical Findings for Non-Technical Partners
A significant challenge is simplifying nuanced AI performance data for stakeholders who may lack technical expertise. Prepare by practicing how to explain the 'why' behind a model's failure without relying on jargon, focusing instead on the business or safety implications.
Maintaining Rigor Under Tight Timelines
The role requires being 'careful and methodical' while working under pressure. Prepare by discussing your strategies for maintaining high-quality documentation and rigorous testing protocols when project deadlines are compressed.
Defining Trust in Emerging Technologies
Since 'trustworthy' AI is subjective and evolving, you may face ambiguity in defining evaluation criteria. Prepare by researching current industry standards for AI safety and fairness, and be ready to discuss how you would adapt these to specific, novel use cases.

Practice your answers

Technical practice

Designing AI Evaluation Frameworks

What to coverExplain your approach to creating a robust evaluation framework for a new AI system. Focus on how you define success metrics, select representative test datasets, and ensure the framework accounts for real-world edge cases.

Conducting Systematic Error Analysis

What to coverDescribe your methodology for performing error analysis on an ML model. Discuss how you categorize failure modes, identify root causes, and translate these findings into actionable improvements for the model or data.

Large Language Model Experimentation

What to coverDetail your experience running structured experiments with LLMs. Address how you manage prompt engineering, control for variability in outputs, and measure performance against specific trust or safety benchmarks.

Data Annotation Quality Control

What to coverHow do you develop clear annotation guidelines to ensure high-quality, consistent labels? Explain your process for monitoring annotator performance and resolving ambiguities in the labeling process.

Python for ML Workflows

What to coverDiscuss how you utilize Python libraries to automate ML evaluation pipelines. Focus on your ability to build modular, reusable code for data processing, model testing, and result visualization.

Assessing AI Trustworthiness and Fairness

What to coverWhat specific metrics or techniques do you use to evaluate the fairness and safety of an AI system? Suggestion: Prepare to discuss trade-offs between model performance and safety constraints.

Building Supporting Tooling

What to coverDescribe a time you built a prototype or tool to support an evaluation task. Explain the problem you were solving, the technologies used, and how the tool improved the efficiency of the research process.

NLP Fundamentals in Evaluation

What to coverHow do you apply NLP concepts to evaluate model performance? Discuss your familiarity with metrics like BLEU, ROUGE, or more modern embedding-based evaluation methods for generative models.

Behavioural practice

Communicating Technical Findings to Non-Technical Stakeholders

What to coverDescribe a situation where you had to explain complex AI evaluation results to a non-technical partner or client. How did you adapt your communication style to ensure they understood the implications and recommendations?

Managing Tight Timelines in Research

What to coverTell me about a time you had to deliver research results under a tight deadline. How did you prioritize your tasks and maintain a methodical approach to ensure the quality of your evaluation?

Collaborating with External Partners

What to coverProvide an example of how you have handled requirement discussions with external project partners. How do you balance their specific needs with the technical constraints of the AI systems you are evaluating?

Working Independently on Complex Problems

What to coverDescribe a project where you were required to work independently to solve a technical challenge. What steps did you take to define your path forward and ensure your work aligned with the broader project goals?

Sharpen your knowledge

Topics to prepare

AI Trustworthiness and Safety Frameworks
Review concepts related to AI fairness, robustness, and safety evaluation. Exercise: Research industry-standard AI evaluation frameworks (e.g., NIST AI RMF or similar) and draft a conceptual framework for assessing a hypothetical LLM for bias or hallucinations. This is relevant to the role's mission of ensuring emerging technologies are trustworthy and safe.
Error Analysis Methodologies
Study systematic approaches to diagnosing model failures, such as slice-based analysis or confusion matrix interpretation. Exercise: Take a small dataset of model predictions and manually categorize errors into types (e.g., data quality, logic, or edge cases) to practice identifying root causes. This directly supports the requirement to conduct error analysis and provide actionable recommendations.

Skills to revise

LLM Experimentation and Evaluation
Practice running structured experiments with LLMs using Python. Exercise: Use a library like LangSmith or Promptfoo to evaluate an LLM's response consistency across different prompts. This is essential for the role's focus on running experiments and building supporting prototypes.
Data Annotation Guideline Development
Develop clear, unambiguous instructions for human annotators. Exercise: Create a set of annotation guidelines for a specific NLP task, such as sentiment analysis or fact-checking, and test them by annotating a small sample to identify potential points of confusion. This aligns with the responsibility to develop guidelines and monitor labelling quality.
Technical Communication for Non-Technical Stakeholders
Practice translating complex ML metrics into business-relevant insights. Exercise: Write a one-page summary of a model's performance report that explains technical findings (e.g., F1-score, latency) in plain language for a non-technical project partner. This is critical for the role's requirement to engage with external clients.

Your preparation checklist

Develop an AI evaluation framework project
Create a portfolio project that demonstrates your ability to design a structured evaluation framework for an LLM. Include a test dataset, a defined scoring methodology, and a report detailing your error analysis and recommendations for model improvement.
Prepare documentation for data annotation
Draft a sample set of annotation guidelines for a specific NLP task. This demonstrates your methodical approach to ensuring labelling quality, which is a core requirement for the role.
Practice explaining technical AI concepts
Prepare to explain complex ML evaluation metrics or LLM failure modes to a non-technical audience. Practice summarizing your technical findings into clear, actionable insights suitable for project partners and clients.
Review Python-based ML evaluation workflows
Refresh your knowledge of Python libraries used for model evaluation and error analysis. Be ready to discuss how you have used these tools in past projects to validate AI system performance.
Formulate questions for the interviewers
Prepare thoughtful questions regarding the Centre's current evaluation priorities, the types of external partners you will engage with, and the specific methodologies used to ensure AI systems are safe and fair in real-world conditions.

Availability last checked: 18 Oct 2026. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy

KEEP EXPLORING

Similar AI jobs

Related skills and specializations in Singapore. Matched to this role, not your profile.

Explore more jobs

Finding similar opportunities…