Senior Vice President, AI/ML Software Engineer

What matters most

• Own the custom agent architecture

  The role owns the technical vision for a multi-agent ecosystem, including orchestration, tool use, planning loops, shared memory and content-policy enforcement. It also makes build-versus-buy decisions across the AI toolchain.

• Deliver the full document-to-answer pipeline

  The posting assigns end-to-end RAG ownership, from OCR and document preprocessing through chunking, embeddings, hybrid retrieval, re-ranking and generation. It also calls for knowledge-graph construction and graph-based retrieval.

• Make AI quality measurable

  Golden-truth datasets, prompt versioning, retrieval and extraction metrics, agent completion rates, hallucination detection, and human-in-the-loop correction are explicit responsibilities. Extraction accuracy matters because outputs affect financial operations.

• Engineer for production and integration

  This is a hands-on role building agent components and contributing to Python/FastAPI AI services and Java/Spring Boot platform services. It owns CI/CD, deployments, observability and audit trails while coordinating with platform, data and infrastructure teams.

• Lead technical decisions and delivery

  The SVP leads a VP-level AI engineer and a broader team of 4–8 developers, with responsibility for mentoring, architecture reviews, sprint planning, the technical backlog and delivery commitments.

Likely challenges to think through

• Choosing what to build in-house

  Preparation hypothesis: A custom orchestration framework offers control but adds maintenance and integration costs. Prepare a decision framework for agent-framework and vector-store build-versus-buy choices, with criteria such as extensibility, reliability and operational burden.

• Balancing autonomy with verifiable extraction

  Preparation hypothesis: Planning and self-correction may improve coverage while making errors harder to trace in financial documents. Prepare to explain how golden-truth tests, confidence scoring, agent traces and human review would constrain unreliable outputs.

• Connecting new AI systems to established platforms

  Preparation hypothesis: Iterating on RAG and agents while maintaining stable enterprise services may create interface and release trade-offs. Prepare an integration approach covering Python–Java API boundaries, schema evolution, deployment gates and observable failure recovery.

• Balancing hands-on work with team ownership

  Preparation hypothesis: Implementing core components while managing a VP-level engineer and 4–8 developers could compete with architecture reviews and delivery planning. Prepare to describe how to delegate implementation, set technical decision boundaries and retain accountability for quality.

Topics to prepare

• Custom multi-agent architecture

  The posting calls for ownership of a custom orchestration framework, including ReAct-style loops, tool use, shared memory and validation. Revise by drawing an agent workflow for document extraction and explaining how agents share state, recover from errors and stop.

• RAG and vectorization trade-offs

  BNY specifies chunking, embedding selection, hybrid search, re-ranking and context assembly. Revise by comparing two retrieval designs for the same document set, stating how you would measure relevance and choose between them.

• Document intelligence

  The role spans raw OCR output through structured extraction and generation. Revise by tracing a document with noisy OCR through preprocessing, extraction, confidence scoring and validation; identify where human correction would enter.

• Evaluation-driven AI

  The posting names golden-truth datasets, MRR, NDCG, extraction F1, agent success rates and hallucination detection. Revise by defining a small labeled test set and specifying what each metric would reveal—and miss.

• Production architecture and governance

  Required experience includes observable AI workloads, error recovery, audit trails and enterprise API patterns. Revise by reviewing a service design for failure handling, circuit breakers, caching, execution traces and auditability. PII handling and data residency are preferred governance knowledge, not stated requirements.

• Senior technical leadership

  The role leads a VP-level engineer and 4–8 developers, with architecture reviews, sprint planning and delivery commitments. Revise by preparing a sample architecture decision: alternatives, build-versus-buy reasoning, review questions, dependencies and a delivery sequence.

Skills to revise

• Build an asynchronous Python AI service

  Strong Python 3.11+, FastAPI, async/await, Poetry, Pydantic and pytest are sought. As practice, implement a validated extraction endpoint with an asynchronous processing path and pytest cases for malformed input and downstream failure.

• Implement an agent loop

  The posting calls for hands-on agent loops, tool registries, persistent memory and reasoning traces. As practice, code a bounded tool-selection loop that records state transitions and tests a failed tool call.

• Assemble a document retrieval pipeline

  The role owns preprocessing, chunk boundaries, metadata, vector indexes and retrieval ranking. As practice, index a small document set, compare chunking and hybrid-search results, and inspect what reaches the generation context.

• Write an evaluation harness

  Golden-truth validation and measurable quality gates are explicit. As practice, run a fixed set of extraction and retrieval cases, calculate extraction F1 and a ranking metric, and flag regressions between two prompt or retrieval versions.

• Integrate Java and Python services

  The posting seeks Java 21, Spring Boot 3.x and microservices alongside Python/FastAPI services. As practice, sketch or implement an API contract between the services, including validation, timeouts and error responses.

• Instrument a production pipeline

  BNY assigns ownership of CI/CD, containerized deployments, agent traces, token tracking and audit pipelines. As practice, add trace identifiers, retrieval-quality telemetry and an audit record to a sample pipeline, then outline its Docker and GitLab CI/CD checks.

• Select a production AI project to walk through

  Prepare a diagram and concise account of your own role, architecture decisions, real workloads, failure recovery, audit trails and measured results. The posting emphasizes production delivery rather than prototypes; use only outcomes you can substantiate.

• Sketch a custom multi-agent architecture

  As interview practice, draw agents, tool registries, planning loops, shared memory, validation and reasoning traces for document extraction. Explain when to build an orchestration component versus use an existing tool; BNY explicitly distinguishes this role from configuring an agent wrapper.

• Trace a document through the RAG pipeline

  Prepare a design walkthrough from OCR and preprocessing to chunking, embeddings, hybrid retrieval, re-ranking and context assembly. Show how you would compare vector-store choices and, if relevant, add entity extraction and graph-based retrieval.

• Draft an evaluation and correction plan

  Outline a golden-truth dataset, prompt-version tests, retrieval metrics such as MRR or NDCG, extraction F1, agent success rates and hallucination checks. Include proposed quality gates and human-in-the-loop correction; these are preparation examples, not stated hiring exercises.

• Prepare a hands-on service design example

  Bring a small code or design example you can explain for a Python/FastAPI agent or document service and its Java 21/Spring Boot integration. Be ready to discuss async processing, Pydantic validation, pytest, APIs and resilience decisions without implying every listed technology was used in your example.

• Document leadership and operational decisions

  Prepare specific examples of architecture reviews, mentoring, backlog trade-offs and cross-team coordination, alongside a deployment sketch covering CI/CD, containers, agent traces, token usage and auditability. The role leads a VP-level engineer and a broader team of 4–8 developers.

• Write targeted questions for BNY

  Ask how extraction quality is measured for financial operations, where custom-agent versus platform boundaries sit, how golden-truth data and human correction are maintained, and which platform, data and infrastructure dependencies are most important. Treat answers as information to learn, not assumptions about current processes.