bytedance / bytedance/bolt

[Feature] Support Spark encode

Open
#925 0 comments 0 reactions 0 assignees View on GitHub
enhancement needs triage
Dominant language
C++
Stars
177
Forks
107
Avg merge
3d 10h
Merged PRs (30d)
49

Description

### Feature Category

SQL Function / Operator

### Problem / Use Case

Apache Spark provides `encode`, and Gluten already maps it
(`Sig[Encode](ENCODE)` in `ExpressionMappings.scala`), but Bolt registers no
`encode`, so the expression falls back.

### Proposed Solution

Implement Spark-compatible `encode`:

```text
encode(str: STRING, charset: STRING) -> BINARY
```

Encodes the string using the named charset. Spark supports at least
`US-ASCII`, `ISO-8859-1`, `UTF-8`, `UTF-16BE`, `UTF-16LE`, `UTF-16`.

Bolt's prestosql registry already has `to_utf8`, which covers the
`charset = 'UTF-8'` case only (and takes no charset argument). A first
increment could map the UTF-8 case and reject or fall back on other charsets;
full support needs real charset conversion.

See also the companion issue for `decode`.

### Reproduction Steps

Tested against the Bolt backend of Gluten, branch
[`add_bolt_backend_new`](https://github.com/taiyang-li/incubator-gluten/tree/add_bolt_backend_new)
(this is the branch the expression mappings referenced below come from).

Point the launcher at that branch and start Spark with Gluten + Bolt:

```bash
export GLUTEN_FORK_URL=https://github.com/taiyang-li/incubator-gluten.git
export GLUTEN_FORK_BRANCH=add_bolt_backend_new
scripts/launch-spark.sh start --build
```

Then run:

```sql
select encode(cast(id as string), 'UTF-8') as r from range(10);
```

Observed physical plan (falls back — plain `Project` fed by `ColumnarToRow`,
instead of `ProjectExecTransformer`):

```text
*(1) Project [encode(cast(id#780L as string), UTF-8) AS r#774]
+- *(1) ColumnarToRow
+- ColumnarRange 0, 10, 1, 4, 10, [id#..L]
```

For contrast, a natively-offloaded expression on the same source produces:

```text
^(-1) ProjectExecTransformer [...]
```

The query still returns correct results — it is a silent performance
regression, not a failure.

### References / Prior Art

- Spark SQL reference: https://spark.apache.org/docs/latest/api/sql/string-functions/#encode
- Apache Spark `Encode`
- Bolt already has `to_utf8` (prestosql registry), covering UTF-8 only

### Acceptance Criteria

- Match Spark for UTF-8 round-trips at minimum.
- Define and test behaviour for unsupported/unknown charset names.
- Cover NULL input.
- Confirm the Gluten plan no longer falls back.

### Importance

Medium (Nice to have)

### Willingness to Contribute

Yes, but I need guidance

Contributor guide

Open the contributing guide

Research direction

Start with the `Sig[Encode](ENCODE)` mapping in `ExpressionMappings.scala` and Bolt’s prestosql registry entry for `to_utf8`. Reproduce the issue with `scripts/launch-spark.sh start --build` and the provided `select encode(...)` query on the Gluten branch. Done means UTF-8 results match Spark, NULL and unsupported charset behavior are tested, and the plan uses `ProjectExecTransformer` instead of falling back.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, spark
Domain
backend, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.