apache / apache/datafusion-comet

Failure-injection tests for native Iceberg writes

Open
#5,646 0 comments 0 reactions 0 assignees View on GitHub
area:Iceberg area:writer enhancement requires-triage
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

The native Iceberg write path has documented failure semantics (mid-task failure retries with attempt-unique file names, partial results are never committed, commit failures reuse `SparkWrite.abort`), and two known gaps (#5618 task-attempt cleanup, #5277 orphans on commit failure). What it does not have is tests that inject failures and assert those semantics hold. `CometIcebergWriteActionSuite` has one commit-time conflict test and one aborted-job test; neither fails inside the native writer, and neither checks what files are left behind.

### Describe the potential solution

Add a failure-injection section to the Iceberg write tests covering, for the native path:

- A task that fails inside iceberg-rust after at least one data file has been finalized (for example an object store that starts rejecting writes after N puts, using the in-memory or a wrapped `file://` store): the retry succeeds, the table commits exactly once, and no file from the failed attempt is referenced by any manifest.
- A task that fails after the native writer returns but before `TaskCommit` (manifest decode or metrics rebuild throws): same assertions, plus the paths from the decoded manifest are deleted once #5618 lands.
- A commit failure after all tasks succeed: the table is unchanged, and once #5277 lands, no data files from the attempt remain.
- Speculative execution: two attempts of the same task both finish; exactly one set of files is committed.
- Executor loss mid-write on a multi-task write: the stage retries and the final row count is exact.

Each test should assert both the visible outcome (rows, snapshot count) and the storage state (files under the data location versus files referenced by manifests), since the orphan gaps are invisible to readers.

### Additional context

Part of the native Iceberg writes epic, #5649. Related: #5618, #5277, #5361 (introduced the failure-handling section in `iceberg-writes.md`).

Contributor guide

Open the contributing guide

Research direction

Start with CometIcebergWriteActionSuite and the failure-handling section introduced in iceberg-writes.md, then review related issues #5618 and #5277 before running the existing Iceberg write tests. Done means native-path failure-injection tests cover task retries, pre-commit failures, commit failures, speculation, and executor loss, asserting rows and snapshots plus data files versus manifest references.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, scala, spark
Domain
data-engineering, databases, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.