Vice President, AI/ML Software Engineer

What matters most

• Build the agent framework

  The role calls for a custom orchestration engine, including agent loops, tool registries, memory, self-correction and multi-agent coordination—not just configuration of an existing framework. Prepare to explain architectural choices and how you would test agent behavior.

• Own RAG from documents to generation

  The posting assigns end-to-end ownership from OCR and ingestion through chunking, embeddings, indexing, hybrid retrieval, re-ranking and context assembly. Prepare to discuss how you would diagnose quality problems at each stage.

• Make quality measurable

  Golden-truth datasets, extraction accuracy, retrieval metrics, regression detection and A/B comparisons are explicit duties. Prepare an approach to evaluating changes before they reach production.

• Deliver governed production services

  The engineer will build FastAPI extraction, validation and classification APIs, contribute to Java/Spring Boot workflow integration, and maintain audit trails and data-governance controls. Prepare to explain service boundaries, testing and operational traceability.

• Lead through implementation and mentorship

  This is a senior individual-contributor role working with an SVP lead, mentoring 2–3 mid-level engineers, reviewing architecture and translating stakeholder needs into technical solutions. Prepare examples of technical decisions you documented and helped others implement.

Likely challenges to think through

• Balancing agent autonomy with control

  Preparation hypothesis, not a confirmed BNY problem: custom multi-agent workflows may gain flexibility at the cost of harder-to-predict behavior. Be ready to describe state boundaries, tool permissions, failure handling and tests for agent loops.

• Tracing extraction errors across the pipeline

  Preparation hypothesis: when accuracy affects financial operations, an incorrect result could originate in OCR, chunking, retrieval or generation. Prepare a staged evaluation and debugging approach that separates those failure sources rather than treating the LLM as the only cause.

• Connecting new AI components to established workflows

  Preparation hypothesis: greenfield AI services that integrate with existing Java/Spring Boot workflows may face trade-offs between iteration speed, API stability and auditability. Prepare to discuss versioned interfaces, validation, audit records and rollout testing without assuming a particular BNY process.

Topics to prepare

• Custom agent orchestration

  Suggested exercise: sketch an agent loop with a tool registry, memory and state, reasoning traces, self-correction, and multi-agent handoffs; explain how it avoids uncontrolled retries. Relevance: BNY explicitly calls for building a custom orchestration engine, not merely configuring a framework.

• End-to-end RAG design

  Suggested exercise: compare section-aware and sliding-window chunking, then explain when metadata filtering, hybrid search, re-ranking, and context assembly improve retrieval. Relevance: the role owns RAG from document ingestion through generation.

• Document extraction and LLM reliability

  Suggested exercise: trace how OCR errors could affect text segmentation, entity extraction, structured output parsing, and downstream validation; propose where to detect failures. Relevance: the posting emphasizes intelligent document processing and extraction accuracy in financial operations.

• Evaluation and quality gates

  Suggested exercise: define a small golden-truth set and explain separate measures for extraction accuracy and retrieval quality, plus a regression and A/B comparison plan. Relevance: BNY explicitly describes evaluation harnesses and measurable quality gates; precision/recall and MRR/NDCG are listed as preferred knowledge.

• Production integration and governance

  Suggested exercise: draw the boundaries between an AI API, a business workflow, metadata and audit records, and content-policy checks; identify decisions to bring to an architecture review. Relevance: the role includes FastAPI services, Java/Spring Boot integration, AI metadata schemas, governance controls, and senior-level design collaboration.

Skills to revise

• Implement an agent loop in Python

  Suggested exercise: code a small Python agent that selects registered tools, persists state across turns, parses a structured response, and records a trace; test a failed tool call and a bounded correction. Relevance: these are explicit agent-pipeline responsibilities, and Python 3.11+ is required.

• Build an incremental retrieval pipeline

  Suggested exercise: take sample OCR text through preprocessing, chunking, embeddings, and a vector index; add a batch load and an incremental update, then inspect retrieval results. Relevance: the posting calls for OCR-to-retrieval ownership and batch and incremental vectorization.

• Expose a validated FastAPI endpoint

  Suggested exercise: implement an async extraction or validation endpoint with Pydantic request and response models, documented errors, and an OpenAPI contract; add tests for invalid input and an LLM timeout. Relevance: FastAPI, async patterns, Pydantic, RESTful API design, and LLM error handling are stated requirements.

• Automate evaluation with pytest

  Suggested exercise: write unit and integration tests plus a golden-truth evaluation that flags extraction or retrieval regressions between two pipeline versions. Relevance: the role explicitly requires testing rigor, retrieval metrics, and automated regression detection.

• Review and document an AI service design

  Suggested exercise: write a short design note covering pipeline-run and audit metadata, a content-policy checkpoint, and an operational runbook; practice explaining trade-offs to an engineer and a business stakeholder. Relevance: the role includes metadata schemas, governance controls, architecture reviews, runbooks, stakeholder collaboration, and mentoring.

• Map evidence to the stated requirements

  Prepare a one-page reference to relevant education, software engineering experience, production AI/ML work, and Python, RAG, agent, and testing examples. The posting specifies a related bachelor’s degree, 6+ years in software engineering, and 2+ years building production AI/ML systems.

• Build an end-to-end RAG project walkthrough

  Select a real project, if available, and diagram document ingestion and OCR post-processing through chunking, embeddings, index updates, hybrid retrieval, re-ranking, context assembly, and generation. Note your decisions, ownership, and measured results; these stages appear in the role.

• Practice a custom agent-framework design

  As preparation, sketch an agent loop with tool registration, function calling, memory and state, reasoning traces, self-correction, and multi-agent coordination. Explain failure handling and why you would build orchestration rather than only configure a framework, as the posting emphasizes.

• Prepare an evaluation plan

  Draft an example golden-truth dataset and quality gates for extraction and retrieval. Show how you would run regression checks and A/B comparisons, using metrics such as extraction precision and recall or retrieval MRR and NDCG where appropriate.

• Outline a production API and governance design

  Sketch a FastAPI extraction or validation endpoint with Pydantic models, async and error-handling choices, pytest coverage, and an integration boundary with a Java/Spring Boot workflow. Include proposed audit trails, AI metadata, content-policy enforcement, and data-governance controls.

• Document senior-level collaboration examples

  Prepare concise accounts, if applicable, of owning a platform component, translating architecture into code, reviewing designs, mentoring engineers, and writing operational runbooks. The role is a senior individual contributor position that works with an SVP lead and mentors 2–3 engineers.

• Write targeted questions for BNY

  Ask how the team defines extraction accuracy and release quality gates, which agent-framework components are greenfield, how AI services integrate with existing workflows, and what ownership boundaries this role would have. These are questions to clarify, not assumptions about current processes.