apache / apache/gluten

[BOLT] backends-bolt is not covered by CI and has drifted from the shared APIs

Open
#12,998 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Description

`backends-bolt` is not built by any CI job. The profile is `activeByDefault=false` (`pom.xml:2085-2088`), `.github/workflows/` has no reference to it, and the one script that does (`dev/gen-all-config-docs.sh:31`) is not wired into a workflow. As a result the module drifted away from the shared APIs without anyone noticing, and by the time #12454 merged it no longer compiled at all: it still called seven `SparkShims` methods removed by #12954 and two symbols removed by #12981, and overrode two `BackendSettingsApi` settings that no longer exist. #12999 repairs those twelve compile errors, but the drift it exposed is wider than the build break.

This issue collects what is left. Everything below is pre-existing and none of it comes from #12999.

### 1. `-Pbackends-bolt -Pdelta` still does not compile

`BoltDeltaComponent.scala:40` calls `OffloadDeltaScan()` with no arguments, but the case class has taken a parameter since it grew deletion-vector support:

```scala
case class OffloadDeltaScan(enableNativeDmlRowIndexScan: Boolean) extends OffloadSingleNode
```

There is no default value, so `mvn test-compile -Pspark-3.4 -Pscala-2.13 -Pbackends-bolt -Piceberg -Piceberg-test -Pdelta` fails with `not enough arguments for method apply`. Velox passes `new VeloxDeltaConfig(c.sqlConf).enableNativeDmlRowIndexScan` (`VeloxDeltaComponent.scala:56-58`); bolt has no `BoltDeltaConfig`, so this needs a decision rather than a mechanical copy. Passing `false` is the conservative option: `shouldFallbackDeletionVectorDmlScan` then falls the scan back rather than offloading it, and since bolt has not compiled with `-Pdelta` since the parameter was added, there is no prior behaviour to preserve.

### 2. Native write drops the session Hadoop conf

`BoltTransformerApi.scala:113` builds the native write parameters from `write.caseInsensitiveOptions` only. Velox first merges the session Hadoop conf (`VeloxTransformerApi.scala:159-165`):

```scala
val writeOptions = Option(write.session).map { session =>
val hadoopConf = session.sessionState.newHadoopConfWithOptions(write.options)
CaseInsensitiveMap(hadoopConf.iterator().asScala.map(e => e.getKey -> e.getValue).toMap)
}.getOrElse(write.caseInsensitiveOptions)
```

`nativeConf` looks up the Parquet writer settings by key, so on bolt anything set through `spark.hadoop.*` never reaches the native writer. Reproduce with `set spark.hadoop.parquet.enable.dictionary=false` followed by an `INSERT`: the file is written with Gluten's defaults instead, and differs from both vanilla Spark and Velox. No error and no fallback, so it is silent. `parquet.block.size`, `parquet.page.size`, `parquet.writer.version` and the codec levels behave the same way.

### 3. Bloom filter rewrite is injected at the wrong point

`BoltRuleApi.scala:77` registers `BloomFilterMightContainJointRewriteRule` at `injectPreTransform`. Velox moved it to `injectPre` for GLUTEN-12013 (`VeloxRuleApi.scala:83`), with a comment explaining why: when `ExpandFallbackPolicy` promotes a stage fallback to a whole-stage fallback, the reversion strips rewrites applied at `injectPreTransform`. That can leave one stage producing Spark-format bloom filter bytes while another still reads them as native-format, which fails with "Unsupported BloomFilter version". `BoltRuleApi.scala:125` injects the same `ExpandFallbackPolicy`, so the path is reachable on bolt too. The fix is the one-word move to `injectPre`.

### 4. Smaller divergences from Velox

- `validateDataTypes` in `BoltBackend.scala:331` inspects only top-level `StructField`s, where Velox uses a recursive `hasUnsupportedType` (`VeloxBackend.scala:362`). Writing `array` passes bolt's check and reaches the native writer; Velox falls back.
- `GenerateTransformStageId()` is never injected. Velox (`VeloxRuleApi.scala:139`) and CH (`CHRuleApi.scala:149`) both inject it, so on bolt `WholeStageTransformer.transformStageId` keeps its default and stage ids in explain output are not unique.
- `BoltIteratorApi.scala:118` has no equivalent of Velox's `needMetadataColumns` short-circuit (`VeloxIteratorApi.scala:124-136`), so a scan with no metadata columns still allocates one empty map per file. Cost only.

### 5. Spark 3.3 leftovers

#12902 removed Spark 3.3 support, but #12454 brought some of it back: `gluten-ut/spark33/src/test/scala/org/apache/gluten/utils/bolt/{BoltSQLQueryTestSettings,BoltTestSettings}.scala`, plus four `spark33/` directories under `backends-bolt/src/test/resources/tpch-approved-plan/` (`v1`, `v1-ras`, `v1-bhj`, `v1-bhj-ras`). Nothing builds `gluten-ut/spark33` any more, so these are dead files.

### 6. gluten-ut only wires bolt for 3.5

`gluten-ut/spark35/src/test/backends-bolt/` exists; `spark40` and `spark41` have only `backends-clickhouse` and `backends-velox`. So `-Pbackends-bolt -Pspark-ut` compiles on 3.4 and 3.5 but not on 4.0 or 4.1, where the shared suites cannot resolve `GlutenColumnarWriteTestSupport` and friends.

### What would stop this recurring

A CI job that compiles `-Pbackends-bolt` on the supported Spark versions would have caught the twelve errors #12999 fixes, and would catch item 1 today. Whether that job also runs bolt's suites is a separate question, since the native library needs Linux plus GCC 10-12 or Clang 16 (`docs/bolt-quick-start.md`).

Happy to split this into separate issues if that suits the bolt owners better.

cc @taiyang-li @FelixYBW @zhztheplayer

Contributor guide

Open the contributing guide

Research direction

Start with the listed backends-bolt files, especially BoltDeltaComponent.scala, BoltTransformerApi.scala, BoltRuleApi.scala, BoltBackend.scala, and BoltIteratorApi.scala, then run the provided Maven test-compile command. Compare the corresponding Velox implementations and inspect the Spark 3.3/4.x test wiring. Done means the supported bolt profiles compile, the identified behavioral divergences are covered, obsolete files are removed, and CI exercises the supported versions.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
backend, build-system, ci-cd
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.