delta-io / delta-io/delta

[BUG][Spark] Spark contradicts delta protocol by enforcing monotonic txn versions

Open
#5,259 0 comments 0 reactions 0 assignees View on GitHub
bug delta-spark
Dominant language
Scala
Stars
9k
Forks
2.2k
Avg merge
1d 20h
Merged PRs (30d)
108

Description

## Bug

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

### Describe the problem

From [the docs on idempotent writes](https://docs.delta.io/delta-batch/#idempotent-writes):

> Any attempt to write data with txnAppId = dailyETL and txnVersion as 23422 or less is ignored because the txnVersion is less than the last recorded txnVersion in the table.

we can also see this is true from a test:
https://github.com/delta-io/delta/blob/4b7e2affebeff8a86a4aabd24a95f9be8af27520/spark/src/test/scala/org/apache/spark/sql/delta/DeltaSuite.scala#L2485

however this contradicts what is in written in [the protocol spec](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#transaction-identifiers):

> The Delta transaction protocol does not, for example, assume monotonicity of the version and it would be valid for the version to decrease, possibly representing a "rollback" of an earlier transaction.

### Environment information

* Delta Lake version: 4.0.0
* Spark version: 4.0.0
* Scala version: 2.13

### Willingness to contribute

The Delta Lake Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Delta Lake code base?

- [ ] Yes. I can contribute a fix for this bug independently.
- [x] Yes. I would be willing to contribute a fix for this bug with guidance from the Delta Lake community.
- [ ] No. I cannot contribute a bug fix at this time.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.