New opportunity

Product Manager, AI (Staff/Principal)

PermitFlow · New York City, United States

About this role

PermitFlow is hiring a Staff- or Principal-level Product Manager, AI to own its core AI infrastructure and workflow agents for construction permitting. The role combines product strategy, UX and technical judgment to improve the speed and accuracy of permit and inspection submissions across complex jurisdictional requirements. It reports to the Head of Product; level depends on experience and interview performance.

Responsibilities

Deliver workflow agents that comprehend construction projects, extract details from documents and support accurate compliance submissions, including forms and inspections.
Define field- and submission-level standards for when results are reliable enough to submit, and own the quality bar governing releases.
Segment use cases to pursue single-shot automation where accuracy permits, while designing assistive agents and tools for permit coordinators when human judgment is needed.
Partner with operations on early testing, golden data sets, evaluations and structured error analysis. Develop user-feedback mechanisms and internal tools for testing and improving agentic systems.
Learn users’ workflows, translate operational insights into product decisions and serve as an applied AI resource for teams with traditional software backgrounds.

Qualifications

7+ years of product management experience, or an equivalent track record in forward-deployed engineering, applied research or operations leadership. The posting does not specify a single years-of-experience minimum across these alternatives.
Experience shipping AI into production where accuracy has real-world consequences, including direct ownership of a shipped AI or agentic system’s quality bar.
Hands-on evaluation experience with golden data sets, LLM-as-judge, human annotation and online monitoring; sound judgment about when an agent is preferable to a simpler product or process change.
Strong technical fluency, close partnership with engineering and operations, and the ability to provide clarity across teams amid ambiguity. The company seeks demonstrated ownership, urgency and high-quality delivery.

Location and compensation: The role is hybrid in New York City, with Monday, Wednesday and Friday in-office; NYC-based candidates or those open to relocation are preferred. Listed compensation is $230K–$285K annually plus equity, varying by location, level, experience and skills. Benefits include company-paid base medical, dental and vision coverage, a healthcare FSA, 401(k), unlimited PTO, paid family leave, a home-office and equipment stipend, in-office meals and commuter benefits.

For full details about this role, click “Apply on company site” to view the employer’s original posting.

Skills for this role

AI product managementAI agentsAgentic infrastructureAgent designAI evaluationGolden data setsLLM-as-judgeHuman annotationOnline monitoringStructured error analysisHuman-in-the-loop systemsWorkflow automationUser experience designProduct instrumentationCross-functional collaborationOperations collaborationConstruction permitting

Checking your profile…

Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.

Understand the role

What matters most

Deliver permitting workflow agents
Role: Own agents that comprehend construction projects and support accurate permit and inspection submissions, including document extraction and form completion. Preparation: Be ready to explain how you would map the end-to-end workflow and measure both speed and submission accuracy.
Set the submission quality bar
Role: Define “reliable enough to submit” at field and submission level and own release gates. Preparation: Outline how you would evaluate consequential errors, set review thresholds and decide whether a system is ready to ship.
Choose where automation belongs
Role: Segment use cases for single-shot submission where accuracy allows, while building assistive tools for permit coordinators where judgment is needed. Preparation: Explain how you would compare full automation, human review and a simpler product or process change.
Build an evaluation and feedback flywheel
Role: Partner with operations on early testing, golden data sets and structured error analysis; build user-feedback mechanisms and internal testing tools. Preparation: Describe how annotations, LLM-as-judge and online monitoring could inform improvements without relying on any one signal alone.
Lead shared agentic infrastructure
Role: Own core AI infrastructure and tools used by other teams, and serve as an applied AI resource to teams with traditional software backgrounds. Preparation: Show how you would turn operational findings into clear product and technical decisions across engineering and operations.

Likely challenges to think through

Variation across jurisdictions
Likely challenge (inference): Different rules, systems and processes across roughly 20,000 jurisdictions may make a reusable agent difficult to reconcile with local requirements. Prepare by outlining how you would segment workflows, validate jurisdiction-specific cases and avoid overgeneralizing from a narrow test set.
Speed versus costly submission errors
Likely challenge (inference): Errors can compound from document extraction through forms, while failed submissions delay projects. Prepare by explaining how field-level checks, submission-level gates and escalation to a person would balance faster processing against error risk.
Improvement signals versus review burden
Likely challenge (inference): Golden data sets, human annotation and user feedback can improve agents but require operational effort and may expose different kinds of error. Prepare by describing how you would prioritize cases for review, analyze failures systematically and check whether changes improve live performance.

Practice your answers

Technical practice

How would you design a workflow agent that turns construction documents into a permit submission?

What to coverSuggested answer guidance: Map document comprehension, project-detail extraction, requirement checks and PDF form completion as separate steps. Explain how you would detect missing or conflicting information before errors compound into a failed submission.

What would make an AI-generated field—and an entire submission—reliable enough to submit?

What to coverSuggested answer guidance: Distinguish field-level correctness from submission-level completeness and consistency, accounting for the cost of a failed submission. Describe how evidence, review requirements and evaluation results would inform a release gate rather than treating one accuracy score as sufficient.

How would you decide which permitting use cases are ready for single-shot automation?

What to coverSuggested answer guidance: Segment cases by variation in jurisdictional requirements, availability of source information and consequences of errors. Explain when you would retain human review and what evidence would justify increasing autonomy.

How should an assistive agent work with a permit coordinator?

What to coverSuggested answer guidance: Describe an interface that presents extracted details, supporting evidence and unresolved questions so a coordinator can exercise judgment efficiently. Address the trade-off between review speed and making consequential mistakes easy to overlook.

How would you build golden data sets with an operations team?

What to coverSuggested answer guidance: Cover representative submissions, clear annotations for expected fields and requirements, and examples of ambiguous or difficult cases. Explain how operations input and versioning would keep the set useful as workflows vary across jurisdictions.

How would you combine LLM-as-judge and human annotation in agent evaluations?

What to coverSuggested answer guidance: Compare their usefulness for scalable checks versus nuanced, consequential errors. Explain how you would check judge outputs against human review and investigate disagreement rather than assuming either method is definitive.

What would you monitor after releasing a permitting agent?

What to coverSuggested answer guidance: Propose instrumentation for submission outcomes, corrections, human interventions and errors by workflow segment. Explain how you would use those signals to detect quality regressions without mistaking reduced review activity for improved accuracy.

How would you turn submission errors into an internal testing and improvement toolset?

What to coverSuggested answer guidance: Classify errors by step and cause, then connect cases to reproducible tests and evaluation trends that teams can inspect. Discuss how you would prioritize fixes by customer impact and verify that changes do not degrade other cases.

How would you design user feedback to improve the agentic data flywheel?

What to coverSuggested answer guidance: Make it easy for users to correct an output and indicate what was wrong in the context of their work. Explain how feedback could become reviewable evaluation data while avoiding the assumption that every correction is unambiguous ground truth.

When would you choose an agent over a simpler product or process change?

What to coverSuggested answer guidance: Compare the workflow's need for adaptive reasoning with the reliability, maintenance and user-effort costs of an agent. Give a decision framework that also considers simpler software or operational changes, as the posting explicitly calls for that judgment.

Behavioural practice

Tell me about a time you owned the quality bar for a consequential AI release.

What to coverSuggested answer guidance: Use a real situation, the release decision you personally made and its result. Explain the evidence and trade-offs you weighed, including how you handled unresolved risks.

Describe a time operations insight changed your product direction.

What to coverSuggested answer guidance: Set out the operational problem, how you worked with frontline colleagues and what you changed. State the observed result and distinguish their input from your own decision.

Give an example of creating clarity across teams when an AI product problem was ambiguous.

What to coverSuggested answer guidance: Describe the uncertainty, your actions to align engineering, operations and product, and the resulting decision or delivery. Include what remained uncertain and how you made it manageable.

Tell me about a time you helped a team with a traditional software background make an applied-AI decision.

What to coverSuggested answer guidance: Use a real example to show how you explained the technical limits or evaluation evidence and adapted the discussion to the team's needs. Describe the action the team took and the result without overstating your individual contribution.

Describe a time you had to move quickly without lowering the quality bar.

What to coverSuggested answer guidance: Identify the deadline or pressure, the scope or safeguards you chose, and the outcome. Explain what you deferred, monitored or escalated to balance urgency with accountability.

Sharpen your knowledge

Topics to prepare

Submission reliability and release gates
The role owns the quality bar at both field and submission level. To revise, sketch how an incorrect extracted field, missing requirement, or incomplete PDF form could affect a submission, then propose separate field-level and submission-level release criteria. The criteria are preparation examples, not stated PermitFlow thresholds.
Automation versus human judgment
The posting calls for single-shot submission where accuracy allows and assistive workflows for permit coordinators otherwise. To revise, compare a routine, well-specified case with one needing judgment; explain when to automate, request review, or use a simpler process change.
Evaluation and data flywheel
Golden data sets, LLM-as-judge, human annotation, online monitoring, and structured error analysis are explicitly named. To revise, map what each would reveal before and after release, and where user feedback would feed the next evaluation cycle.
Permitting workflow and compounding errors
The role covers construction-document comprehension, detail extraction, requirement checks, PDF forms, permits, and inspections across differing jurisdictions. To revise, draw the sequence from documents to submission and identify where an early mistake could propagate.
Agent infrastructure and product tradeoffs
The role owns core AI infrastructure and workflow agents and calls for technical fluency with engineering. To revise, outline a conceptual agent workflow with checkpoints, feedback, and testing; explain which parts should be shared across use cases and which depend on a specific workflow. This outline is a preparation suggestion, not a prescribed architecture.

Skills to revise

Design a golden-set evaluation
As a practice exercise, create a small hypothetical set of permit-field examples, record expected answers, and specify how human annotations would resolve disagreements. This rehearses the hands-on evaluation work explicitly requested; it is not a PermitFlow data set.
Analyze errors by failure mode
Using invented extraction and submission mistakes, build an error table showing the affected field, downstream effect, severity, and proposed correction. The posting explicitly calls for structured error analysis ahead of submission.
Specify a coordinator review flow
Sketch an assistive workflow showing what a permit coordinator sees, corrects, and confirms before submission, including how feedback is captured. This practices the posted responsibility to design human-in-the-loop tools and usable agent feedback.
Plan online quality monitoring
Draft a monitoring plan that separates field corrections, submission failures, and user-reported issues, with a response for a deteriorating signal. Online monitoring and ownership of a shipped system’s quality bar are explicit qualifications; the signals are suggested examples.
Translate operations findings into decisions
Practice turning a hypothetical operations observation into a concise product decision: the user problem, whether an agent or simpler change fits, the test, and the release risk. This reflects the role’s emphasis on operations partnership, user workflows, and cross-functional clarity.

Your preparation checklist

Map the permitting workflow
Prepare a one-page map of document comprehension, detail extraction, requirement checks, PDF forms, and permit or inspection submission. Mark where errors could compound; the posting identifies failed submissions as costly.
Select a production AI walkthrough
If you have a relevant shipped system, prepare a concise account of your decisions, quality-bar ownership, engineering and operations partnerships, and measured outcomes. Distinguish your own work from the team’s.
Draft a submission quality gate
As a practice exercise, define how you would assess reliability at both the field and complete-submission levels, handle uncertain results, and decide whether a release is ready. State your assumptions rather than presenting thresholds as PermitFlow policy.
Sketch an evaluation and feedback loop
Prepare a sample plan covering golden data, human annotation, LLM-as-judge checks, online monitoring, and structured error analysis. Show how early testing with operations and user feedback would inform improvements.
Compare automation with assisted workflows
Work through a hypothetical permitting use case: identify when single-shot submission might be appropriate, when a coordinator should review work, and when a simpler product or process change would beat an agent. Sketch the coordinator’s review experience.
Prepare a cross-functional leadership example
Choose an example, if available, that shows how you translated operational insight into a product decision, resolved technical ambiguity with engineering, and gave other teams a clear path to delivery.
Write questions for the Head of Product and team
Ask which use cases are closest to full automation, how submission quality is assessed today, where coordinators most need assistance, and what evaluation or internal-tooling gaps this role should prioritize.

Availability last checked: Nov 4, 2026. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy

KEEP EXPLORING

Similar AI jobs

Related skills and specializations in United States. Matched to this role, not your profile.

Explore more jobs

Finding similar opportunities…