[VL] Gluten do not respect the order that WriteFiles operator requires
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
WriteFiles operator requires a specific order that is guaranteed by Sort operator added by V1Writes rule. If the ordering of child operator like SMJ satisfies the order, Sort will not be needed. However, Gluten could transform SMJ to SHJ, which can disrupt the order. And EnsureLocalSortRequirements rule can't ensure the order because WriteFiles operator lacks the requiredChildOrdering property. As a result, there is a risk of data loss.
Here is an example :
vanilla spark plan
```
+- == Final Plan ==
Execute InsertIntoHiveTable
+- WriteFiles
+- Project
+- SortMergeJoin [dynamic_partition_column#186]
```
gluten plan
```
+- == Final Plan ==
Execute InsertIntoHiveTable
+- WriteFiles
+- VeloxColumnarToRow
+- ProjectExecTransformer
+- ShuffledHashJoinExecTransformer [dynamic_partition_column#186]
```
Potential solutions:
1. workaround: Disable `spark.sql.optimizer.plannedWrite.enabled` to move the logic of ensuring the order requirement after Gluten plan transformation.
2. Perhaps, WriteFiles should have the requiredChildOrdering property.
3. EnsureLocalSortRequirements rule supports ensuring ordering for WriteFiles.
### Gluten version
_No response_
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Assessment
This issue has not been assessed yet.