apache / apache/gluten

[VL] GlutenDataFrameSuite support ToPrettyString binary formats

Open
#11,570 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Description

https://github.com/apache/incubator-gluten/pull/11566
The version spark 3.5 works well, but spark4.0 and Spark 4.1 introduces BINARY_OUTPUT_STYLE, we only support the default HEX_DISCRETE as spark 3.5, so I will update the test `getRows: binary` after spark 4.0.

```
val BINARY_OUTPUT_STYLE = buildConf("spark.sql.binaryOutputStyle")
.doc("The output style used display binary data. Valid values are 'UTF-8', " +
"'BASIC', 'BASE64', 'HEX', and 'HEX_DISCRETE'.")
.version("4.0.0")
.enumConf(BinaryOutputStyle)
.createOptional
```

### Gluten version

None

Contributor guide

Open the contributing guide

Research direction

Start with the GlutenDataFrameSuite test named `getRows: binary` and inspect how it handles Spark 3.5's default HEX_DISCRETE output. Compare that behavior with Spark 4.0 and 4.1's BINARY_OUTPUT_STYLE values, then update the test coverage so the expected binary-format behavior is validated.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
testing-qa
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.