Override reference.conf to Support gs FileSystem
- Dominant language
- Scala
- Stars
- 1.5k
- Forks
- 315
- PR merge metrics
- No merged PRs in 30d
Description
I am using 0.11 version of MLeap and Trying to add support for GS
I checked bundle-hdfs/src/main/resources/reference.conf and found that we can use other schemes like S3 to store the MLeap Bundle in S3. I wanted to add support for Google Storage (GS) scheme.
I created new reference.conf in the server and added it to spark classpath using `--conf spark.executor.extraClassPath="/home/myuser/reference.conf"`. I get below error:
```
Exception in thread "main" java.util.NoSuchElementException: key not found: gs
at scala.collection.MapLike$class.default(MapLike.scala:228)
at scala.collection.AbstractMap.default(Map.scala:59)
at scala.collection.MapLike$class.apply(MapLike.scala:141)
at scala.collection.AbstractMap.apply(Map.scala:59)
at ml.combust.bundle.BundleRegistry.fileSystemForUri(BundleRegistry.scala:146)
at ml.combust.bundle.BundleWriter$$anonfun$save$5.apply(BundleWriter.scala:51)
at ml.combust.bundle.BundleWriter$$anonfun$save$5.apply(BundleWriter.scala:50)
at scala.util.Success$$anonfun$map$1.apply(Try.scala:237)
at scala.util.Try$.apply(Try.scala:192)
at scala.util.Success.map(Try.scala:237)
at ml.combust.bundle.BundleWriter.save(BundleWriter.scala:49)
```
NOTE : I changed the `HadoopBundleFileSystem` which can default to `Seq("hdfs", "gs")` and it worked. I need help in finding a way to propagate the settings via reference.conf with no code change. Any help is appreciated.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.