Rebase vendored Substrait proto from v0.23.0 to 0.98.0
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Background
Gluten vendors its Substrait `.proto` files (`gluten-substrait/src/main/resources/substrait/proto/substrait/`) pinned near Substrait **v0.23.0** with local patches, while upstream Substrait is now at **0.98.0**. The JVM plan producer and both native consumers (Velox, ClickHouse) regenerate from this single proto source and ship together, and Gluten's Substrait plans are transient (built per query, serialized across JNI, discarded) — so there is **no wire-compatibility constraint** and the rebase is a pure code migration.
### Goal
Bring the vendored proto to official 0.98.0 core (verbatim) plus Gluten's genuinely engine-specific additions grafted at non-colliding numbers, migrating the JVM producer and both native backends in lockstep.
### Approach
Land as small, independently-green PRs, one per message family (each keeps `protoc` + JVM + both native builds compiling). Minimal rebase first — relocate only colliding fork fields and drop dead/decided-drop items; extracting engine-specific forks to formal Substrait extensions is deferred to a follow-up.
### Increments
One PR per message family; each keeps `protoc` + JVM + both native builds green. **Progress: 11 merged · Inc 8 to do (ReadRel split into 7a/7b/7c; 7a merged, 7b in review, 7c in review as two stacked PRs) · 1 resolved upstream.**
- [x] **0** — Remove unused vendored proto files + dead proto-based type derivation + `Expression.Enum` — merged (#12598)
- [x] **1a** — `plan.proto` / `extensions.proto` → 0.98 (URI→URN) — merged (#12604)
- [x] **1b** — `AdvancedExtension.optimization` singular → repeated — merged (#12642)
- [x] **2** — Type system: temporal → `PrecisionTimestamp*`; relocate `Type.Nothing` (`kind` 33→40, kept) — merged (#12650)
- [x] **3** — `CrossRel` → `NestedLoopJoinRel` (+ join-type enum remap) — merged (#12700)
- [x] **4** — Windowing: `WindowRel` → `ConsistentPartitionWindowRel`; `WindowType` → `BoundsType` — merged (#12727)
- [x] **5a** — `FetchRel` offset/count → expressions — merged (#12704)
- [x] **5b** — `AggregateRel.Grouping` → expression references — merged (#12724)
- [x] **5c** — `ExpandRel.advanced_extension` — converged upstream via [substrait-io/substrait#1167](https://github.com/substrait-io/substrait/pull/1167) (merged, ships 0.100.0); Gluten's graft already matches, no Gluten PR needed
- [x] **6a** — `TopNRel` → `Expression count` + `offset` + `FetchMode` — merged (#12728)
- [x] **6b** — `WriteRel`: relocate `bucket_spec` (→ 1000); add `create_mode` / `advanced_extension`; `OUTPUT_MODE_MODIFIED_TUPLES` → `_RECORDS` — merged (#12746)
- [ ] **7** — `ReadRel` (the largest vendored message) — split into three independent slices (disjoint proto regions and consumers, can land in any order):
- [x] **7a** — `VirtualTable.values` → `Expression.Nested.Struct expressions` — merged (#12849)
- [ ] **7b** — text read-options redesign: `TextReadOptions` → `DelimiterSeparatedTextReadOptions` (map `header`/`null_value` to the 0.98 fields; graft `max_block_size`/`empty_as_default` at 1000+) — PR #12917 (in review)
- [ ] **7c** — `read_type` → verbatim 0.98, in two stacked slices:
- [ ] **7c-i** — add official `IcebergTable iceberg_table = 9`; relocate the `stream_kafka` graft off the field-9 collision → 1000 — PR #12832 (in review)
- [ ] **7c-ii** — delete the `stream_kafka` graft entirely (making `read_type` verbatim 0.98) and remodel Gluten's ClickHouse Kafka read onto the official `extension_table = 8` path: the `StreamKafka` payload moves to a new Gluten-owned `kafka.proto`, packed into `ReadRel.ExtensionTable.detail` and discriminated by the `Any` type_url (the pattern MergeTree/Range should converge onto — see #12842) — PR #12841 (in review)
- [ ] **8** — `Rel` oneof reconciliation to exact 0.98 layout + drop `RelRoot.output_schema` — to do
- [ ] **Docs (final)** — after all increments land, reconcile `docs/developers/SubstraitModifications.md` with the finished proto: bump the stated base version v0.23.0 → 0.98.0 and drop entries for messages that ended up verbatim-upstream (e.g. `ReadRel.VirtualTable`), leaving only the genuinely remaining Gluten deviations
### Upstream Substrait convergence PRs
Some kept forks converge by landing the equivalent field/type in Substrait core (`substrait-io/substrait`) rather than by dropping the fork. These are tracked under the [`apache-gluten`](https://github.com/substrait-io/substrait/pulls?q=label%3Aapache-gluten) label:
- [x] [substrait-io/substrait#1167](https://github.com/substrait-io/substrait/pull/1167) — `advanced_extension` on `ExpandRel` — merged, ships 0.100.0 → drives increment 5c; Gluten's graft already matches verbatim
- [ ] [substrait-io/substrait#1147](https://github.com/substrait-io/substrait/pull/1147) — core `Type.Null` for untyped nulls — open → increment 2 relocates `Type.Nothing` to `kind` 40 to match; converge by renaming `Nothing`→`Null` once released (see note below)
- [ ] [substrait-io/substrait#1138](https://github.com/substrait-io/substrait/pull/1138) — JSON read options on `ReadRel.LocalFiles` — open → upstreams Gluten's `json` read-options fork; increment 7b keeps the vendored `json = 15` until this merges
- [ ] [substrait-io/substrait#1139](https://github.com/substrait-io/substrait/pull/1139) — partition fields on `ReadRel.LocalFiles` — open → upstreams Gluten's partition-columns fork; the field-16 collision with the `iceberg` fork is a post-merge concern
- ~~[substrait-io/substrait#1140](https://github.com/substrait-io/substrait/pull/1140)~~ — docs on typing untyped nulls — closed, superseded by [substrait-io/substrait#1147](https://github.com/substrait-io/substrait/pull/1147)
Tracked for later extraction waves (not driving a rebase increment):
- [ ] [substrait-io/substrait#917](https://github.com/substrait-io/substrait/pull/917) — `GenerateRel` for lateral view / unnest (@EpsilonPrime) — upstreams the pattern of Gluten's `GenerateRel` fork
- [x] [substrait-io/substrait#1105](https://github.com/substrait-io/substrait/pull/1105) — expression-valued window-aggregate bounds (@yongchul) — merged, ships 0.102.0; future convergence for Gluten's window frame bounds, mirroring the FetchRel offset→expression change
### Note on `Type.Nothing`
Following the Substrait community sync (2026-07-29), the earlier plan to *drop* the `Type.Nothing` fork is superseded: upstream is adding a dedicated core `null` type ([substrait-io/substrait#1147](https://github.com/substrait-io/substrait/pull/1147) — `Type.Null` at `kind` field 40). Increment 2 relocates `Nothing` to field 40 to match, so once that PR releases the fork converges by **renaming the message** `Nothing`→`Null` — no field renumbering, no consumer-mapping changes. Gluten keeps `Type.Nothing` until then.
Related: upstream consolidation work is tracked under the `apache-gluten` label in `substrait-io/substrait`.
Contributor guide
Research direction
Start in gluten-substrait/src/main/resources/substrait/proto/substrait/ and review the remaining ReadRel and Rel increments, including the referenced in-review PRs. Check that protoc, the JVM producer, and both native consumers build together after each message-family change. Done means all increments land and docs/developers/SubstraitModifications.md reflects the final 0.98 proto and remaining Gluten deviations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, scala
- Domain
- backend
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100