apache / apache/gluten

Rebase vendored Substrait proto from v0.23.0 to 0.98.0

Open
#12,597 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Background

Gluten vendors its Substrait `.proto` files (`gluten-substrait/src/main/resources/substrait/proto/substrait/`) pinned near Substrait **v0.23.0** with local patches, while upstream Substrait is now at **0.98.0**. The JVM plan producer and both native consumers (Velox, ClickHouse) regenerate from this single proto source and ship together, and Gluten's Substrait plans are transient (built per query, serialized across JNI, discarded) — so there is **no wire-compatibility constraint** and the rebase is a pure code migration.

### Goal

Bring the vendored proto to official 0.98.0 core (verbatim) plus Gluten's genuinely engine-specific additions grafted at non-colliding numbers, migrating the JVM producer and both native backends in lockstep.

### Approach

Land as small, independently-green PRs, one per message family (each keeps `protoc` + JVM + both native builds compiling). Minimal rebase first — relocate only colliding fork fields and drop dead/decided-drop items; extracting engine-specific forks to formal Substrait extensions is deferred to a follow-up.

### Increments

One PR per message family; each keeps `protoc` + JVM + both native builds green. **Progress: 11 merged · Inc 8 to do (ReadRel split into 7a/7b/7c; 7a merged, 7b in review, 7c in review as two stacked PRs) · 1 resolved upstream.**

- [x] **0** — Remove unused vendored proto files + dead proto-based type derivation + `Expression.Enum` — merged (#12598)
- [x] **1a** — `plan.proto` / `extensions.proto` → 0.98 (URI→URN) — merged (#12604)
- [x] **1b** — `AdvancedExtension.optimization` singular → repeated — merged (#12642)
- [x] **2** — Type system: temporal → `PrecisionTimestamp*`; relocate `Type.Nothing` (`kind` 33→40, kept) — merged (#12650)
- [x] **3** — `CrossRel` → `NestedLoopJoinRel` (+ join-type enum remap) — merged (#12700)
- [x] **4** — Windowing: `WindowRel` → `ConsistentPartitionWindowRel`; `WindowType` → `BoundsType` — merged (#12727)
- [x] **5a** — `FetchRel` offset/count → expressions — merged (#12704)
- [x] **5b** — `AggregateRel.Grouping` → expression references — merged (#12724)
- [x] **5c** — `ExpandRel.advanced_extension` — converged upstream via [substrait-io/substrait#1167](https://github.com/substrait-io/substrait/pull/1167) (merged, ships 0.100.0); Gluten's graft already matches, no Gluten PR needed
- [x] **6a** — `TopNRel` → `Expression count` + `offset` + `FetchMode` — merged (#12728)
- [x] **6b** — `WriteRel`: relocate `bucket_spec` (→ 1000); add `create_mode` / `advanced_extension`; `OUTPUT_MODE_MODIFIED_TUPLES` → `_RECORDS` — merged (#12746)
- [ ] **7** — `ReadRel` (the largest vendored message) — split into three independent slices (disjoint proto regions and consumers, can land in any order):
- [x] **7a** — `VirtualTable.values` → `Expression.Nested.Struct expressions` — merged (#12849)
- [ ] **7b** — text read-options redesign: `TextReadOptions` → `DelimiterSeparatedTextReadOptions` (map `header`/`null_value` to the 0.98 fields; graft `max_block_size`/`empty_as_default` at 1000+) — PR #12917 (in review)
- [ ] **7c** — `read_type` → verbatim 0.98, in two stacked slices:
- [ ] **7c-i** — add official `IcebergTable iceberg_table = 9`; relocate the `stream_kafka` graft off the field-9 collision → 1000 — PR #12832 (in review)
- [ ] **7c-ii** — delete the `stream_kafka` graft entirely (making `read_type` verbatim 0.98) and remodel Gluten's ClickHouse Kafka read onto the official `extension_table = 8` path: the `StreamKafka` payload moves to a new Gluten-owned `kafka.proto`, packed into `ReadRel.ExtensionTable.detail` and discriminated by the `Any` type_url (the pattern MergeTree/Range should converge onto — see #12842) — PR #12841 (in review)
- [ ] **8** — `Rel` oneof reconciliation to exact 0.98 layout + drop `RelRoot.output_schema` — to do
- [ ] **Docs (final)** — after all increments land, reconcile `docs/developers/SubstraitModifications.md` with the finished proto: bump the stated base version v0.23.0 → 0.98.0 and drop entries for messages that ended up verbatim-upstream (e.g. `ReadRel.VirtualTable`), leaving only the genuinely remaining Gluten deviations

### Upstream Substrait convergence PRs

Some kept forks converge by landing the equivalent field/type in Substrait core (`substrait-io/substrait`) rather than by dropping the fork. These are tracked under the [`apache-gluten`](https://github.com/substrait-io/substrait/pulls?q=label%3Aapache-gluten) label:

- [x] [substrait-io/substrait#1167](https://github.com/substrait-io/substrait/pull/1167) — `advanced_extension` on `ExpandRel` — merged, ships 0.100.0 → drives increment 5c; Gluten's graft already matches verbatim
- [ ] [substrait-io/substrait#1147](https://github.com/substrait-io/substrait/pull/1147) — core `Type.Null` for untyped nulls — open → increment 2 relocates `Type.Nothing` to `kind` 40 to match; converge by renaming `Nothing`→`Null` once released (see note below)
- [ ] [substrait-io/substrait#1138](https://github.com/substrait-io/substrait/pull/1138) — JSON read options on `ReadRel.LocalFiles` — open → upstreams Gluten's `json` read-options fork; increment 7b keeps the vendored `json = 15` until this merges
- [ ] [substrait-io/substrait#1139](https://github.com/substrait-io/substrait/pull/1139) — partition fields on `ReadRel.LocalFiles` — open → upstreams Gluten's partition-columns fork; the field-16 collision with the `iceberg` fork is a post-merge concern
- ~~[substrait-io/substrait#1140](https://github.com/substrait-io/substrait/pull/1140)~~ — docs on typing untyped nulls — closed, superseded by [substrait-io/substrait#1147](https://github.com/substrait-io/substrait/pull/1147)

Tracked for later extraction waves (not driving a rebase increment):

- [ ] [substrait-io/substrait#917](https://github.com/substrait-io/substrait/pull/917) — `GenerateRel` for lateral view / unnest (@EpsilonPrime) — upstreams the pattern of Gluten's `GenerateRel` fork
- [x] [substrait-io/substrait#1105](https://github.com/substrait-io/substrait/pull/1105) — expression-valued window-aggregate bounds (@yongchul) — merged, ships 0.102.0; future convergence for Gluten's window frame bounds, mirroring the FetchRel offset→expression change

### Note on `Type.Nothing`

Following the Substrait community sync (2026-07-29), the earlier plan to *drop* the `Type.Nothing` fork is superseded: upstream is adding a dedicated core `null` type ([substrait-io/substrait#1147](https://github.com/substrait-io/substrait/pull/1147) — `Type.Null` at `kind` field 40). Increment 2 relocates `Nothing` to field 40 to match, so once that PR releases the fork converges by **renaming the message** `Nothing`→`Null` — no field renumbering, no consumer-mapping changes. Gluten keeps `Type.Nothing` until then.

Related: upstream consolidation work is tracked under the `apache-gluten` label in `substrait-io/substrait`.

Contributor guide

Open the contributing guide

Research direction

Start in gluten-substrait/src/main/resources/substrait/proto/substrait/ and review the remaining ReadRel and Rel increments, including the referenced in-review PRs. Check that protoc, the JVM producer, and both native consumers build together after each message-family change. Done means all increments land and docs/developers/SubstraitModifications.md reflects the final 0.98 proto and remaining Gluten deviations.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, scala
Domain
backend
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.