[BUG][Spark] Spark contradicts delta protocol by enforcing monotonic txn versions
- Dominant language
- Scala
- Stars
- 9k
- Forks
- 2.2k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 108
Description
## Bug
#### Which Delta project/connector is this regarding?
- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)
### Describe the problem
From [the docs on idempotent writes](https://docs.delta.io/delta-batch/#idempotent-writes):
> Any attempt to write data with txnAppId = dailyETL and txnVersion as 23422 or less is ignored because the txnVersion is less than the last recorded txnVersion in the table.
we can also see this is true from a test:
https://github.com/delta-io/delta/blob/4b7e2affebeff8a86a4aabd24a95f9be8af27520/spark/src/test/scala/org/apache/spark/sql/delta/DeltaSuite.scala#L2485
however this contradicts what is in written in [the protocol spec](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#transaction-identifiers):
> The Delta transaction protocol does not, for example, assume monotonicity of the version and it would be valid for the version to decrease, possibly representing a "rollback" of an earlier transaction.
### Environment information
* Delta Lake version: 4.0.0
* Spark version: 4.0.0
* Scala version: 2.13
### Willingness to contribute
The Delta Lake Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Delta Lake code base?
- [ ] Yes. I can contribute a fix for this bug independently.
- [x] Yes. I would be willing to contribute a fix for this bug with guidance from the Delta Lake community.
- [ ] No. I cannot contribute a bug fix at this time.
Contributor guide
Assessment
This issue has not been assessed yet.