MERGE INTO doesn't work when using non-deterministic expressions in Pyspark
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.7.1
### Query engine
Spark
### Please describe the bug 🐞
When I try to use a MERGE INTO statement to write to an Iceberg table in a Glue Job that applies a non-deterministic expression (e.g:`uuid`) I always get `[INVALID_NON_DETERMINISTIC_EXPRESSIONS] The operator expects a deterministic expression`. Is this something that is addressed in newer Iceberg versions?
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
No files, tests, or entry points are named. Start by reproducing the MERGE INTO case with a non-deterministic expression such as uuid in Spark on Iceberg 1.7.1, then compare behavior with newer Iceberg versions; done means the supported behavior is fixed and covered by a regression test, or the limitation is clearly established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python, spark
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100