combust / combust/mleap

Override reference.conf to Support gs FileSystem

Open
#550 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.5k
Forks
315
PR merge metrics
No merged PRs in 30d

Description

I am using 0.11 version of MLeap and Trying to add support for GS

I checked bundle-hdfs/src/main/resources/reference.conf and found that we can use other schemes like S3 to store the MLeap Bundle in S3. I wanted to add support for Google Storage (GS) scheme.

I created new reference.conf in the server and added it to spark classpath using `--conf spark.executor.extraClassPath="/home/myuser/reference.conf"`. I get below error:

```
Exception in thread "main" java.util.NoSuchElementException: key not found: gs
at scala.collection.MapLike$class.default(MapLike.scala:228)
at scala.collection.AbstractMap.default(Map.scala:59)
at scala.collection.MapLike$class.apply(MapLike.scala:141)
at scala.collection.AbstractMap.apply(Map.scala:59)
at ml.combust.bundle.BundleRegistry.fileSystemForUri(BundleRegistry.scala:146)
at ml.combust.bundle.BundleWriter$$anonfun$save$5.apply(BundleWriter.scala:51)
at ml.combust.bundle.BundleWriter$$anonfun$save$5.apply(BundleWriter.scala:50)
at scala.util.Success$$anonfun$map$1.apply(Try.scala:237)
at scala.util.Try$.apply(Try.scala:192)
at scala.util.Success.map(Try.scala:237)
at ml.combust.bundle.BundleWriter.save(BundleWriter.scala:49)
```
NOTE : I changed the `HadoopBundleFileSystem` which can default to `Seq("hdfs", "gs")` and it worked. I need help in finding a way to propagate the settings via reference.conf with no code change. Any help is appreciated.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.