Hadoop file system options doesn't work
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
The issue happens when running this Delta unit test with Gluten/Velox:
https://github.com/delta-io/delta/blob/074ce173bb070ed60f10e98d3ef7dcc04cc2a744/core/src/test/scala/org/apache/spark/sql/delta/DeltaDataFrameHadoopOptionsSuite.scala#L56
I created a smaller repro using Parquet format:
```
test("test123") {
def myTest() = {
val fakeFileSystemOptions = Map(
"fs.myfake.impl" -> classOf[MyFakeFileSystem].getName,
"fs.myfake.impl.disable.cache" -> "true"
)
val dir = new File("/tmp/test/")
val path = s"myfake://${dir.getCanonicalPath}"
spark.range(1, 10)
.write
.format("parquet")
.mode("overwrite")
.options(fakeFileSystemOptions)
.save(path)
spark.read.format("parquet").options(fakeFileSystemOptions).load(path).foreach(_ => {})
}
withSQLConf("spark.gluten.enabled" -> "false") {
myTest()
}
withSQLConf("spark.gluten.enabled" -> "true") {
//This one fails
myTest()
}
}
import org.apache.hadoop.fs.RawLocalFileSystem
import java.net.URI
object MyFakeFileSystem {
val scheme = "myfake"
val uri = URI.create(s"$scheme:///")
}
/** A fake file system to test whether session Hadoop configuration will be picked up. */
// Can't be an inner class
class MyFakeFileSystem extends RawLocalFileSystem {
override def getScheme: String = MyFakeFileSystem.scheme
override def getUri: URI = MyFakeFileSystem.uri
}
```
Delta: 2.4
Spark 3.4
### Spark version
Spark-3.3.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
_No response_
Contributor guide
Research direction
Start by running DeltaDataFrameHadoopOptionsSuite.scala, then reduce the supplied Parquet repro and compare the spark.gluten.enabled=false and true paths. Trace how the fake filesystem options are passed through the Gluten/Velox path; done means the session Hadoop configuration is honored with Gluten enabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, scala
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100