Senior Vice President, AI / Machine Learning Software Engineer

What matters most

• Own the custom agent architecture

  The role calls for a custom orchestration framework, not merely configuring an existing one. It covers planning and reasoning loops, tool registries, shared memory, inter-agent communication, content-policy controls, and self-correcting extraction agents.

• Deliver document intelligence and RAG end to end

  Ownership runs from OCR and document preprocessing through chunking, embeddings, vector indexing, hybrid retrieval, re-ranking, and context assembly. The role also includes constructing knowledge graphs from extracted entities and relationships.

• Make AI quality measurable

  The posting calls for golden-truth datasets, automated regression and quality gates, retrieval metrics, extraction accuracy, agent task-completion rates, hallucination detection, and human-in-the-loop correction.

• Ship operable systems into enterprise platforms

  This is production AI rather than prototype work. Responsibilities include Python/FastAPI AI services, Java/Spring Boot platform services, CI/CD, containerized deployment, observability, error recovery, audit trails, and coordination with platform, data, and infrastructure teams.

• Lead while remaining hands-on

  The SVP sets technical direction, runs architecture reviews and delivery planning, mentors a team of 4–8 engineers, and directly manages a VP-level AI engineer. The posting also expects implementation of core agent, embedding, scoring, and validation components.

Likely challenges to think through

• Balancing agent autonomy with control

  Preparation hypothesis: self-correcting agents may improve extraction while making decisions harder to reproduce or audit. Prepare a design rationale for bounded tool use, validation, execution traces, human correction, and failure recovery; the trade-off is inferred, while these mechanisms are named in the posting.

• Improving retrieval without obscuring the cause

  Preparation hypothesis: changes to OCR, chunking, embeddings, hybrid search, and re-ranking can interact, making quality gains difficult to attribute. Prepare an evaluation approach using golden-truth documents, retrieval metrics such as MRR and NDCG, extraction F1, and regression checks before changing multiple stages at once.

• Combining greenfield AI with established services

  Preparation hypothesis: a new orchestration framework must evolve without disrupting enterprise integration or delivery commitments. Prepare to explain build-versus-buy criteria, service boundaries between Python and Java, and how architecture reviews and cross-team dependency planning would guide incremental delivery.

Topics to prepare

• Custom multi-agent architecture

  Posting: BNY calls for a custom orchestration framework with planning loops, tool use, shared memory, inter-agent communication and content-policy controls. Revision exercise: sketch an extraction-to-validation agent flow and explain how it handles failed tools, conflicting results and self-correction.

• End-to-end RAG design

  Posting: the role owns RAG from document chunking and embedding selection through vector indexing, hybrid retrieval, re-ranking and context assembly. Revision exercise: compare two chunking and retrieval designs for an unstructured document, explaining dimensionality, ranking and context-window trade-offs.

• Document intelligence and graph retrieval

  Posting: OCR, structured extraction, entity-relationship mapping and knowledge graph construction are part of the work; graph-based retrieval is preferred. Revision exercise: trace how text from an OCR output would become validated fields and linked entities, identifying where extraction errors could propagate.

• Evaluation and feedback

  Posting: BNY specifies golden-truth datasets, MRR, NDCG, extraction F1, agent success rates, hallucination detection and human correction. Revision exercise: define what each metric would measure for one document-extraction pipeline and how corrected results would enter a regression set.

• Production architecture and governance

  Posting: observable, recoverable and auditable AI systems are required, alongside API, caching, circuit-breaker and event-driven patterns; PII handling and data residency are preferred knowledge. Revision exercise: review a hypothetical pipeline failure and specify recovery, tracing and audit decisions, noting where sensitive data requires controls.

• Senior technical leadership

  Posting: this hands-on SVP role leads a VP-level engineer and a team of 4–8, owns architecture reviews and delivery, and coordinates with platform, data and infrastructure teams. Revision exercise: prepare a design-review agenda that weighs build-versus-buy choices, evaluation gates, dependencies and delivery risks.

Skills to revise

• Implement a Python AI service

  Posting: strong Python 3.11+, FastAPI, async/await, Poetry, Pydantic and pytest are required. Revision exercise: build a small asynchronous extraction-and-validation endpoint with typed request and response models, dependency management, and tests for malformed input and service errors.

• Build agent components

  Posting: hands-on work includes agent loops, tool registries, memory persistence, reasoning traces and scoring or validation. Revision exercise: implement a bounded tool-use loop that records decisions and validates an extracted field before returning it.

• Construct a retrieval pipeline

  Posting: hands-on responsibilities span preprocessing, chunk-boundary detection, metadata enrichment and vector index management. Revision exercise: process a small document set into indexed chunks, then compare retrieved passages before and after a chunking or re-ranking change.

• Create evaluation tests

  Posting: the role develops golden-truth validation and retrieval, extraction and agent-completion metrics. Revision exercise: write a small labeled test set and calculate MRR, NDCG and extraction F1; add a regression test for an incorrect extracted value.

• Integrate Java platform services

  Posting: Java 21, Spring Boot 3.x and microservices are required, and the role contributes to platform services. Revision exercise: define and implement a Spring Boot API boundary for an AI service, including a timeout or circuit-breaker behavior and a test of the failure path.

• Operate and observe a deployment

  Posting: the role owns CI/CD, containerized deployments, environment promotion, agent traces, token tracking, retrieval-quality metrics and audit trails; Docker and GitLab CI/CD appear in the stack. Revision exercise: outline or assemble a Docker and GitLab CI/CD workflow with automated tests, then specify the trace and audit fields needed to investigate a failed extraction.

• Map the required experience to evidence

  The posting calls for 10+ years in software engineering and 3+ years leading or technically mentoring teams. Prepare a concise timeline and examples showing scope, technical decisions, and outcomes, without stretching any experience.

• Build a production-AI project walkthrough

  Prepare one architecture diagram and a short account of a document-intelligence system you can discuss: OCR or extraction, validation, failure recovery, observability, audit trails, and your specific contribution. Use measured outcomes where available.

• Practice designing a custom agent framework

  Sketch an extraction pipeline with planning loops, tool registries, shared memory, inter-agent communication, self-correction, and content-policy controls. Be ready to explain build-versus-buy choices; the posting emphasizes building a framework rather than merely configuring one.

• Work through the full RAG path

  Prepare a design exercise from document preprocessing and chunking to embeddings, vector indexing, hybrid retrieval, re-ranking, and context assembly. Explain trade-offs and, if relevant to your experience, how extracted entities could support graph-based retrieval.

• Define measurable quality gates

  Draft a small evaluation plan using golden-truth documents, retrieval metrics such as MRR and NDCG, extraction F1, agent task completion, hallucination checks, and human correction. Show how results would inform regression tests and prompt or retrieval changes.

• Review implementation and operating boundaries

  Prepare a service diagram connecting Python/FastAPI AI components with Java 21/Spring Boot platform services. Annotate async processing, API contracts, caching or circuit breakers, CI/CD, container deployment, execution traces, token tracking, and audit records—areas named in the posting.

• Prepare senior-level leadership examples

  Select examples of architecture reviews, mentoring, sprint planning, delivery trade-offs, and coordination with platform, data, or infrastructure teams. For each, note the decision you owned, how others contributed, and the outcome; the role leads a VP-level engineer and a broader 4–8-person team.

• Bring questions about scope and success measures

  Ask which document workflows are first in scope, what evaluation baselines and human-review paths exist, where custom orchestration must integrate with established platforms, and how architecture and delivery responsibilities are divided across teams. These are questions to clarify, not assumptions about BNY’s current systems.