[Feature] Support Spark encode
- Dominant language
- C++
- Stars
- 177
- Forks
- 107
- Avg merge
- 3d 10h
- Merged PRs (30d)
- 49
Description
### Feature Category
SQL Function / Operator
### Problem / Use Case
Apache Spark provides `encode`, and Gluten already maps it
(`Sig[Encode](ENCODE)` in `ExpressionMappings.scala`), but Bolt registers no
`encode`, so the expression falls back.
### Proposed Solution
Implement Spark-compatible `encode`:
```text
encode(str: STRING, charset: STRING) -> BINARY
```
Encodes the string using the named charset. Spark supports at least
`US-ASCII`, `ISO-8859-1`, `UTF-8`, `UTF-16BE`, `UTF-16LE`, `UTF-16`.
Bolt's prestosql registry already has `to_utf8`, which covers the
`charset = 'UTF-8'` case only (and takes no charset argument). A first
increment could map the UTF-8 case and reject or fall back on other charsets;
full support needs real charset conversion.
See also the companion issue for `decode`.
### Reproduction Steps
Tested against the Bolt backend of Gluten, branch
[`add_bolt_backend_new`](https://github.com/taiyang-li/incubator-gluten/tree/add_bolt_backend_new)
(this is the branch the expression mappings referenced below come from).
Point the launcher at that branch and start Spark with Gluten + Bolt:
```bash
export GLUTEN_FORK_URL=https://github.com/taiyang-li/incubator-gluten.git
export GLUTEN_FORK_BRANCH=add_bolt_backend_new
scripts/launch-spark.sh start --build
```
Then run:
```sql
select encode(cast(id as string), 'UTF-8') as r from range(10);
```
Observed physical plan (falls back — plain `Project` fed by `ColumnarToRow`,
instead of `ProjectExecTransformer`):
```text
*(1) Project [encode(cast(id#780L as string), UTF-8) AS r#774]
+- *(1) ColumnarToRow
+- ColumnarRange 0, 10, 1, 4, 10, [id#..L]
```
For contrast, a natively-offloaded expression on the same source produces:
```text
^(-1) ProjectExecTransformer [...]
```
The query still returns correct results — it is a silent performance
regression, not a failure.
### References / Prior Art
- Spark SQL reference: https://spark.apache.org/docs/latest/api/sql/string-functions/#encode
- Apache Spark `Encode`
- Bolt already has `to_utf8` (prestosql registry), covering UTF-8 only
### Acceptance Criteria
- Match Spark for UTF-8 round-trips at minimum.
- Define and test behaviour for unsupported/unknown charset names.
- Cover NULL input.
- Confirm the Gluten plan no longer falls back.
### Importance
Medium (Nice to have)
### Willingness to Contribute
Yes, but I need guidance
Contributor guide
Research direction
Start with the `Sig[Encode](ENCODE)` mapping in `ExpressionMappings.scala` and Bolt’s prestosql registry entry for `to_utf8`. Reproduce the issue with `scripts/launch-spark.sh start --build` and the provided `select encode(...)` query on the Gluten branch. Done means UTF-8 results match Spark, NULL and unsupported charset behavior are tested, and the plan uses `ProjectExecTransformer` instead of falling back.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, spark
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100