How to read parquet file with Arrow on m1 macs
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
this is the scala code i am using to read parquet
```
val options = new ScanOptions(/*batchSize*/ 100)
val allocator = new RootAllocator
val datasetFactory = new FileSystemDatasetFactory(allocator, NativeMemoryPool.getDefault,
FileFormat.PARQUET, firstFile.toURI.toString)
val dataset = datasetFactory.finish
val scanner = dataset.newScan(options)
```
My gradle is
```
implementation group: 'org.apache.arrow', name: 'arrow-vector', version: '9.0.0'
implementation group: 'org.apache.arrow', name: 'arrow-memory', version: '9.0.0'
runtimeOnly 'org.apache.arrow:arrow-memory-netty:9.0.0'
implementation 'org.apache.arrow:arrow-format:9.0.0'
implementation 'org.apache.arrow:arrow-memory-core:9.0.0'
implementation 'org.apache.arrow:arrow-flight:9.0.0'
implementation 'org.apache.arrow:arrow-dataset:9.0.0'
implementation 'org.apache.arrow:arrow-tools:9.0.0'
```
Error
```
Exception in thread "main" java.lang.UnsatisfiedLinkError: /private/var/folders/rm/_bdtrttd6gb4bxf666vlt57r0000gp/T/jnilib-7676089916881629931.tmp: dlopen(/private/var/folders/rm/_bdtrttd6gb4bxf666vlt57r0000gp/T/jnilib-7676089916881629931.tmp, 0x0001): tried: '/private/var/folders/rm/_bdtrttd6gb4bxf666vlt57r0000gp/T/jnilib-7676089916881629931.tmp' (mach-o file, but is an incompatible architecture (have (x86_64), need (arm64e)))
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1950)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1832)
at java.lang.Runtime.load0(Runtime.java:811)
at java.lang.System.load(System.java:1088)
at org.apache.arrow.dataset.jni.JniLoader.load(JniLoader.java:88)
at org.apache.arrow.dataset.jni.JniLoader.loadRemaining(JniLoader.java:73)
at org.apache.arrow.dataset.jni.JniLoader.ensureLoaded(JniLoader.java:60)
at org.apache.arrow.dataset.jni.NativeMemoryPool.getDefault(NativeMemoryPool.java:34)
at demo.ArrowParquetReader$.delayedEndpoint$demo$ArrowParquetReader$1(ArrowParquetReader.scala:23)
at demo.ArrowParquetReader$delayedInit$body.apply(ArrowParquetReader.scala:14)
at scala.Function0.apply$mcV$sp(Function0.scala:39)
at scala.Function0.apply$mcV$sp$(Function0.scala:39)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
at scala.App.$anonfun$main$1(App.scala:76)
at scala.App.$anonfun$main$1$adapted(App.scala:76)
at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
at scala.collection.AbstractIterable.foreach(Iterable.scala:926)
```
Contributor guide
Research direction
Start with the stack-trace entry points in ArrowParquetReader.scala:23, NativeMemoryPool.getDefault, and org.apache.arrow.dataset.jni.JniLoader.load. Check how the Arrow 9.0.0 dependencies select the native library for the reported arm64e environment; done means the Parquet reader starts without the UnsatisfiedLinkError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100