apache / apache/gluten

Hadoop file system options doesn't work

Open
#4,963 6 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

The issue happens when running this Delta unit test with Gluten/Velox:

https://github.com/delta-io/delta/blob/074ce173bb070ed60f10e98d3ef7dcc04cc2a744/core/src/test/scala/org/apache/spark/sql/delta/DeltaDataFrameHadoopOptionsSuite.scala#L56

I created a smaller repro using Parquet format:

```
test("test123") {
def myTest() = {
val fakeFileSystemOptions = Map(
"fs.myfake.impl" -> classOf[MyFakeFileSystem].getName,
"fs.myfake.impl.disable.cache" -> "true"
)

val dir = new File("/tmp/test/")
val path = s"myfake://${dir.getCanonicalPath}"

spark.range(1, 10)
.write
.format("parquet")
.mode("overwrite")
.options(fakeFileSystemOptions)
.save(path)

spark.read.format("parquet").options(fakeFileSystemOptions).load(path).foreach(_ => {})
}

withSQLConf("spark.gluten.enabled" -> "false") {
myTest()
}

withSQLConf("spark.gluten.enabled" -> "true") {
//This one fails
myTest()
}
}

import org.apache.hadoop.fs.RawLocalFileSystem
import java.net.URI

object MyFakeFileSystem {
val scheme = "myfake"
val uri = URI.create(s"$scheme:///")
}

/** A fake file system to test whether session Hadoop configuration will be picked up. */
// Can't be an inner class
class MyFakeFileSystem extends RawLocalFileSystem {
override def getScheme: String = MyFakeFileSystem.scheme
override def getUri: URI = MyFakeFileSystem.uri
}
```

Delta: 2.4
Spark 3.4

### Spark version

Spark-3.3.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by running DeltaDataFrameHadoopOptionsSuite.scala, then reduce the supplied Parquet repro and compare the spark.gluten.enabled=false and true paths. Trace how the fake filesystem options are passed through the Gluten/Velox path; done means the session Hadoop configuration is honored with Gluten enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, scala
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.