internetarchive / internetarchive/Sparkling
`s3a` URLs don't work in `WarcLoader` (`Wrong FS: s3a://...`)
- Dominant language
- Scala
- Stars
- 17
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
EDIT: this helped with `Wrong FS`, more tickets incoming ;)
```sc.hadoopConfiguration.set("fs.defaultFS", "s3a://commoncrawl/")```
Hey folks, I'm trying to read some common crawl data from S3... See https://github.com/archivesunleashed/aut/issues/556 where I'm using the `aut` pattern, but I get the same symptom using Sparkling by itself:
```
bin/spark-shell --jars ~/dev/Sparkling/target/scala-2.12/sparkling-assembly-0.3.8-SNAPSHOT.jar --packages com.amazonaws:aws-java-sdk:1.12.662,org.apache.hadoop:hadoop-aws:3.3.4
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://192.168.0.73:4040
Spark context available as 'sc' (master = local[*], app id = local-1708561219560).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.4.2
/_/
Using Scala version 2.12.17 (OpenJDK 64-Bit Server VM, Java 11.0.21)
Type in expressions to have them evaluated.
Type :help for more information.
scala> import org.archive.webservices.sparkling._, org.archive.webservices.sparkling.warc._, org.archive.webservices.sparkling.io._
import org.archive.webservices.sparkling._
import org.archive.webservices.sparkling.warc._
import org.archive.webservices.sparkling.io._
scala> val warcs = WarcLoader.load("s3a://commoncrawl/crawl-data/CC-MAIN-2023-50/segments/1700679518883.99/wat/CC-MAIN-20231211210408-20231212000408-00881.warc.wat.gz")
24/02/21 16:20:25 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
java.lang.IllegalArgumentException: Wrong FS: s3a://commoncrawl/crawl-data/CC-MAIN-2023-50/segments/1700679518883.99/wat/CC-MAIN-20231211210408-20231212000408-00881.warc.wat.gz, expected: file:///
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:807)
at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:105)
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:774)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:1100)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:769)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:462)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:115)
at org.apache.hadoop.fs.Globber.doGlob(Globber.java:349)
at org.apache.hadoop.fs.Globber.glob(Globber.java:202)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:2124)
at org.archive.webservices.sparkling.io.HdfsIO.files(HdfsIO.scala:163)
at org.archive.webservices.sparkling.util.RddUtil$.loadFilesLocality(RddUtil.scala:74)
at org.archive.webservices.sparkling.util.RddUtil$.loadBinary(RddUtil.scala:125)
at org.archive.webservices.sparkling.warc.WarcLoader$.load(WarcLoader.scala:56)
... 53 elided
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.