apache / apache/gluten

[VL] Gluten do not respect the order that WriteFiles operator requires

Open
#9,789 6 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

WriteFiles operator requires a specific order that is guaranteed by Sort operator added by V1Writes rule. If the ordering of child operator like SMJ satisfies the order, Sort will not be needed. However, Gluten could transform SMJ to SHJ, which can disrupt the order. And EnsureLocalSortRequirements rule can't ensure the order because WriteFiles operator lacks the requiredChildOrdering property. As a result, there is a risk of data loss.

Here is an example :

vanilla spark plan
```
+- == Final Plan ==
Execute InsertIntoHiveTable
+- WriteFiles
+- Project
+- SortMergeJoin [dynamic_partition_column#186]
```

gluten plan
```
+- == Final Plan ==
Execute InsertIntoHiveTable
+- WriteFiles
+- VeloxColumnarToRow
+- ProjectExecTransformer
+- ShuffledHashJoinExecTransformer [dynamic_partition_column#186]
```

Potential solutions:
1. workaround: Disable `spark.sql.optimizer.plannedWrite.enabled` to move the logic of ensuring the order requirement after Gluten plan transformation.
2. Perhaps, WriteFiles should have the requiredChildOrdering property.
3. EnsureLocalSortRequirements rule supports ensuring ordering for WriteFiles.

### Gluten version

_No response_

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.