Data Engineer Interview Questions: 10 Detailed Answers and Scenarios

Common mistake: Using creation time as if it also represents every later modification.

3. How do you distinguish duplicates from legitimate similar records?

Answer: Choose identity from the source's stable keys and documented business rules, not from title similarity alone. Preserve source IDs and provenance, then define what constitutes an update versus a separate record. Normalize carefully and test collisions. Similar titles can represent different openings, while tracking parameters can make the same underlying URL appear different.

Hypothetical scenario: Two openings share a title and company but have different requisition IDs and locations.

Follow-up: Which evidence would justify merging rather than retaining both?

Common mistake: Deleting all similar-looking rows without a defensible identity rule.

4. How do you prevent readers from seeing a half-published dataset?

Answer: Stage and validate a complete candidate output before making it visible. Use an atomic commit, version pointer or database transaction appropriate to the storage system. Define the publication boundary and retain the previous valid version for recovery. Successful writing of some files is not equivalent to successful publication of a coherent dataset.

Hypothetical scenario: A dashboard reads the new dimension table while its associated fact table still comes from yesterday's run.

Follow-up: Which versioning approach would let readers select a consistent snapshot?

Common mistake: Publishing each intermediate output immediately without a consistency boundary.

5. What makes a schema change backward compatible?

Answer: Compatibility depends on actual reader expectations, not only whether a column was added or removed. Optional additions may be safe, while renamed fields, changed units or new enum values can break consumers. Inventory readers, test representative contracts and coordinate migration where necessary. Version semantic changes explicitly rather than relying on permissive parsing to conceal them.

Hypothetical scenario: A new status value passes storage validation but an older consumer treats every unfamiliar value as closed.

Follow-up: How would you test consumers before enabling that value?

Common mistake: Calling every additive change safe without checking reader behaviour.

6. How do event time and processing time differ?

Answer: Event time represents when something happened according to the source, while processing time represents when your system handles it. Arrival can be delayed or reordered. Choose aggregation windows and correction policies based on the business question, preserve relevant timestamps and account for clock quality. A late event should not silently become an event that happened today.

Hypothetical scenario: An offline device uploads yesterday's measurements during this morning's processing run.

Follow-up: Which timestamp should determine a daily activity report?

Common mistake: Grouping by ingestion date when the metric is defined by occurrence date.

7. When should a pipeline quarantine rather than discard a record?

Answer: Quarantine when a record cannot safely enter the trusted output but can be retained for diagnosis or correction under the applicable data policy. Store the reason, source identity and enough context to replay it safely. Distinguish recoverable data-quality problems from security or retention constraints that require different handling. Monitor quarantine volume so failures do not become invisible data loss.

Hypothetical scenario: A job record contains an invalid location code but otherwise has a valid source identifier.

Follow-up: What information is needed to correct and replay it?

Common mistake: Dropping invalid records silently and reporting the batch as entirely successful.

8. Why does a successful scheduled run not prove freshness?

Answer: A run may complete while the upstream source is stale, pagination is incomplete or no new records were actually consumed. Measure source and output timestamps, expected volumes and checkpoint movement alongside process status. Define freshness relative to the business requirement. Alert on missing progress or delayed data rather than treating an exit code as a completeness guarantee.

Hypothetical scenario: The pipeline reports success but has published the same source snapshot for three days.

Follow-up: Which independent signal would expose the lack of fresh input?

Common mistake: Using scheduler success as the only data-health metric.

9. How would you run a safe historical backfill?

Answer: Define a bounded time range, source snapshot and target identity rules. Estimate resource impact, isolate or throttle the backfill and validate outputs before publication. Make replay idempotent and protect current data from older corrections overwriting newer state. Reconcile counts and changed records, and retain a rollback plan rather than treating a historical rerun as harmless.

Hypothetical scenario: A backfill of last month's records runs alongside today's incremental ingestion.

Follow-up: How would you prevent older versions from replacing newer updates?

Common mistake: Replaying history into production without version ordering or resource limits.

10. When should a consumer advance its checkpoint?

Answer: Advance only when the corresponding output is durably recorded under the chosen delivery semantics. If output and checkpoint cannot share a transaction, use deduplication or an idempotent sink and explain the recovery window. Advancing first can lose records; writing first can replay them after a crash. Correctness requires handling that trade-off explicitly rather than assuming failures occur between batches.

Hypothetical scenario: The process saves its offset and crashes before committing the transformed records.

Follow-up: Which records are at risk, and how could the design prevent loss?

Common mistake: Committing offsets early to make the dashboard appear up to date.

Turn practice into evidence

Implement an ingestion exercise using synthetic job records with stable IDs, late corrections and duplicates. Keep a raw input log, a quarantine table and a published snapshot. Interrupt processing between stages, replay a batch and verify that totals, identities and correction history remain consistent.

Score each response informally: 0 if you cannot explain it, 1 for a definition, 2 for an approach with checks, and 3 for justified trade-offs plus a failure-recovery plan. This is a self-study rubric, not a validated hiring assessment. Revisit weak answers and never invent project outcomes or experience.

Career roadmap and related jobs

Build the underlying skills with the Data Engineer career guide. For a shorter starting point, read AI interview answer frameworks.

Explore relevant live jobs. Listings are filtered by Data Engineering for AI and can include adjacent roles. Read the employer's requirements and original posting before applying. Use one listing to select the questions most relevant to its responsibilities.