New opportunity

Senior AI Engineer

Aedifice · NYC, United States

About this role

Aedifice is hiring a Senior AI Engineer to own significant parts of Wren, an agent runtime that reads building records, drafts filings and narratives, cites sources, and prepares work for licensed professional review. The role emphasizes careful, verifiable work rather than chatbot-style responses. It is based in NYC and is hybrid by default; fully remote work may be considered for the right candidate.

Responsibilities

Write production TypeScript and Python, design tool interfaces for Claude-based agents, and connect retrieval to a growing corpus of statutes and filings.
Own the draft-and-review loop in the first six months, from initial retrieval and tool use through intermediate critique to the format a licensed reviewer expects.
Improve citation accuracy so claims trace to statute sections, prior filings, or observations about a building. Refine drafts for professional use and address cases where an agent could confidently produce incorrect work.
Design evaluations resembling professional exams, develop ways for the runtime to handle uncertainty and defer to a human, and contribute to internal prompt and tool-use regression tooling.
Write and defend design documents before implementation. Work with the research lead, founding engineer, and licensed professionals, including attending line-by-line reviews of draft filings.

Requirements

Seven or more years of software experience, including at least one year of serious work on production LLM-based systems. Candidates should have built dependable software, debugged tool-use loops, developed a thoughtful approach to evaluation, and be able to write clear design documents. The team values legible designs and care with work that another professional must countersign.

Preferred

Experience in a regulated domain, legal tech, medical records, or building systems. Familiarity with Anthropic’s Claude models and the Managed Agents pattern is useful but not required.

Compensation and benefits include competitive cash, early-stage equity on a standard four-year schedule, health and dental coverage, a 401(k) with match, and five weeks of paid time off. An office in lower Manhattan is available.

For full details about this role, click “Apply on company site” to view the employer’s original posting.

Skills for this role

TypeScriptPythonLLM-based systemsAI agentsClaudeTool useRetrievalCitation accuracyAI evaluationPrompt regression testingTechnical writing

Checking your profile…

Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.

Understand the role

What matters most

Own the draft-and-review loop
In the first six months, the role owns the path from retrieval and agent tool use through intermediate critique to a filing draft in the format a licensed reviewer expects.
Make every claim traceable
Harden citations so claims point to statute sections, prior filings, or observations about a building, while improving drafts for professional use and closing paths to confidently incorrect output.
Build boundaries for human judgment
Shape when Wren should not draft, how it flags conditions it cannot assess, and how it presents open questions to a licensed professional.
Evaluate professional-quality work
Design evaluations closer to professional exams than standard ML benchmarks, and contribute to prompt and tool-use regression tooling that catches quiet failures before release.
Deliver legible production designs
Write production TypeScript and Python, design Claude-agent tool interfaces, and connect retrieval to statutes and filings. The role calls for design documents defended in prose and close work with the research lead, founding engineer, and licensed reviewers.

Likely challenges to think through

Evidence gaps versus a complete draft
A likely difficulty is that retrieved records may not support every proposed claim. Prepare to explain a method for checking provenance, withholding unsupported assertions, and leaving a clear question for the reviewer rather than filling a gap with plausible text.
Agent initiative versus reviewer control
The runtime must do substantial drafting without making judgments a licensed professional needs to make. Prepare an example of how you would set stop-or-escalate criteria and present uncertainty in language useful during line-by-line review.
Incremental quality versus shipping speed
Improving citations and practitioner-ready prose may involve small gains that are hard to measure, while tool or prompt changes can introduce quiet regressions. Prepare to describe evaluation cases, regression checks, and how you would defend a measured design before implementation.

Practice your answers

Technical practice

How would you design Wren’s draft-and-review loop from retrieval to licensed review?

What to coverSuggested preparation: Describe the stages named in the posting—retrieval, tool use, intermediate critique, and reviewer-ready output—and what each stage should pass to the next. Explain how you would keep sources and unresolved questions visible throughout.

How would you retrieve relevant statutes and prior filings for a draft?

What to coverSuggested preparation: Discuss how you would identify the applicable material, distinguish authoritative statutes from prior examples, and check that retrieved passages support their intended use. Explain what you would do when relevant material is missing or conflicting.

How would you verify that every material claim has a sound citation?

What to coverSuggested preparation: Address claims grounded in statute sections, prior filings, or on-building observations, as specified in the posting. Explain how you would detect a citation that points to a real source but does not support the claim.

How would you design tool interfaces for a Claude-based agent?

What to coverSuggested preparation: Explain how clear inputs, outputs, and failure states could help the agent use retrieval and drafting tools reliably. Discuss the trade-off between giving an agent flexibility and making its actions inspectable.

How would you debug an agent tool-use loop that produces a confidently wrong filing draft?

What to coverSuggested preparation: Walk through how you would trace the retrieved material, tool calls, intermediate outputs, and final claims to locate the failure. Explain how you would distinguish a retrieval problem from a tool-interface or drafting problem.

What should an intermediate critique catch before a draft reaches a licensed reviewer?

What to coverSuggested preparation: Consider unsupported claims, missing conditions, inconsistent source use, and language that would need substantial professional rewriting. Explain which issues could be corrected in the loop and which should become explicit questions for review.

When should the runtime decline to draft and ask a human?

What to coverSuggested preparation: Describe how you would handle insufficient evidence or a building condition the agent cannot assess, both concerns identified in the posting. Explain how to state the uncertainty as a useful, practitioner-facing question rather than conceal it in a polished draft.

How would you evaluate this system beyond accuracy on a held-out set?

What to coverSuggested preparation: Propose exam-like cases that test source support, judgment under uncertainty, and the usefulness of the package presented for licensed review. Explain how reviewer assessment could reveal failures that a single aggregate score misses.

How would you catch quiet prompt and tool-use regressions before release?

What to coverSuggested preparation: Discuss repeatable cases covering citations, tool choices, drafting, and deferral, reflecting the regression tooling described in the posting. Explain how you would investigate a changed result when a prompt or tool interface changes.

What would you put in a design document before changing the production runtime?

What to coverSuggested preparation: Lay out the problem, proposed interfaces, alternatives, failure modes, and evidence that would justify the change. Explain the trade-off between a clever design and one that another engineer and a licensed reviewer can readily inspect.

Behavioural practice

Tell me about a time you incorporated detailed expert review into a technical product.

What to coverSuggested preparation: Use a real situation, your actions, and the result; distinguish what the expert identified from what you changed. The posting describes line-by-line reviews with licensed professionals, so emphasize how you handled precise corrections.

Describe a time you defended or revised a design in writing before implementation.

What to coverSuggested preparation: Give the situation, the reasoning and feedback you addressed, and the resulting decision. Focus on making trade-offs legible rather than claiming that your initial proposal was necessarily right.

Tell me about a production failure in a system people depended on that you had to diagnose.

What to coverSuggested preparation: Explain the situation, your diagnostic actions, and the outcome without inventing a success story. If relevant to your actual experience, discuss how tracing a tool-use loop changed your understanding of the failure.

Describe a time you slowed or limited a release because the evidence was insufficient.

What to coverSuggested preparation: State the situation, the decision you made, how you communicated it, and the result. Connect your reasoning to the posting’s emphasis on careful work that a professional may need to countersign.

Tell me about a time you coordinated across engineering and domain experts to own a workflow.

What to coverSuggested preparation: Use a real situation, your specific actions, and the result, including how you resolved differing priorities. The role works with a research lead, founding engineer, and licensed professionals, so focus on how you kept responsibilities and open questions clear.

Sharpen your knowledge

Topics to prepare

Evidence and citation provenance
Posting: Wren must tie claims to statute sections, prior filings, or on-building observations. Suggested revision exercise: take a short mock filing and classify each claim by its supporting source; identify claims that lack adequate support.
The draft-and-review loop
Posting: the role owns the path from retrieval and tool use through intermediate critique to a licensed reviewer’s expected format. Suggested revision exercise: diagram that path and explain where critique should change, block, or annotate a draft.
Uncertainty and human deferral
Posting: the runtime must decide when not to draft and how to raise conditions it cannot assess. Suggested revision exercise: compare three hypothetical evidence gaps and state whether to draft, flag a limitation, or ask the licensed professional a specific question.
Evaluation beyond a single accuracy score
Posting: evaluations should resemble professional exams, and quiet prompt or tool-use regressions must be caught. Suggested revision exercise: outline a scoring rubric for citation support, omissions, unsafe certainty, and usefulness to a reviewer.
Agent tool-use failure modes
Posting: the engineer will design interfaces for Claude-based agents and is expected to have debugged production tool-use loops. Suggested revision exercise: reason through a hypothetical loop that repeatedly retrieves irrelevant material, identifying observable symptoms and possible interface or control-flow fixes.

Skills to revise

Production TypeScript and Python
Posting: day-to-day work includes production code in both languages. Suggested practice: implement a small, legible source-record interface in each language, with error handling and tests for missing or malformed records.
Tool-interface design and debugging
Posting: the role designs Claude-based agent tools and debugs tool-use loops. Suggested practice: specify a retrieval tool’s inputs, outputs, and failure cases, then walk through a mock execution trace and pinpoint why a call failed.
Retrieval-backed citation checks
Posting: retrieval spans statutes and filings, and every draft claim needs traceable support. Suggested practice: use mock source excerpts to map draft claims to exact supporting passages; mark unsupported claims for revision or human review.
Regression-case construction
Posting: the engineer contributes to prompt and tool-use regression tooling. Suggested practice: write a small set of repeatable test cases covering a correct citation, an unsupported claim, and an appropriate deferral; define what would count as a regression.
Design documents and review discussion
Posting: designs are written and defended before implementation, with feedback from the research lead, founding engineer, and licensed professionals. Suggested practice: draft a one-page design for an uncertainty handoff, stating tradeoffs, failure cases, and questions to resolve in review.
Practitioner-facing draft revision
Posting: drafts must suit licensed review, and professionals examine filings line by line. Suggested practice: edit a mock draft to distinguish supported statements, unresolved questions, and items requiring professional judgment, then explain each edit.

Your preparation checklist

Prepare a production-system walkthrough
The role calls for production TypeScript and Python and experience with LLM-based systems. Prepare one example, if available, showing architecture, tool-use debugging, reliability decisions, and your specific contributions.
Draft a sample design document
Aedifice expects a design defended before implementation. Outline a proposed draft-and-review loop from retrieval and tool use through intermediate critique to a licensed reviewer’s handoff; label it as a hypothetical design.
Practice a citation-tracing exercise
The role requires claims to trace to statute sections, prior filings, or building observations. Prepare a small hypothetical draft with a source for each claim and a clear response to missing or conflicting evidence.
Outline an evaluation and regression plan
Aedifice describes evaluations resembling professional exams and prompt and tool-use regression tooling. Prepare sample test cases, review criteria, failure categories, and a way to detect quiet regressions.
Define human-escalation decisions
The runtime must sometimes decline to draft, flag conditions it cannot assess, or raise questions for a licensed professional. Prepare examples of each decision and the information a reviewer would need.
Prepare for line-by-line review
Licensed professionals review draft filings closely. Bring an example, if available, of how you incorporated expert feedback into a dependable system; otherwise, prepare a hypothetical revision with an explicit change log.
Write focused questions for the team
Ask how licensed reviewers judge draft readiness, which citation failures are hardest to catch, and how the research lead and founding engineer divide decisions about the draft-and-review loop.
Assemble the requested application materials
The posting asks for a resume and a one-page note explaining why this role and why Aedifice, sent with the subject line “Senior AI Engineer.” Keep the note specific to verifiable work and licensed review.

Availability last checked: Nov 4, 2026. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy

KEEP EXPLORING

Similar AI jobs

Related skills and specializations in United States. Matched to this role, not your profile.

Explore more jobs

Finding similar opportunities…