apache / apache/arrow

How to read parquet file with Arrow on m1 macs

Open
#14,051 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

this is the scala code i am using to read parquet

```
val options = new ScanOptions(/*batchSize*/ 100)

val allocator = new RootAllocator
val datasetFactory = new FileSystemDatasetFactory(allocator, NativeMemoryPool.getDefault,
FileFormat.PARQUET, firstFile.toURI.toString)
val dataset = datasetFactory.finish
val scanner = dataset.newScan(options)

```

My gradle is

```
implementation group: 'org.apache.arrow', name: 'arrow-vector', version: '9.0.0'
implementation group: 'org.apache.arrow', name: 'arrow-memory', version: '9.0.0'
runtimeOnly 'org.apache.arrow:arrow-memory-netty:9.0.0'
implementation 'org.apache.arrow:arrow-format:9.0.0'
implementation 'org.apache.arrow:arrow-memory-core:9.0.0'
implementation 'org.apache.arrow:arrow-flight:9.0.0'
implementation 'org.apache.arrow:arrow-dataset:9.0.0'
implementation 'org.apache.arrow:arrow-tools:9.0.0'

```


Error

```

Exception in thread "main" java.lang.UnsatisfiedLinkError: /private/var/folders/rm/_bdtrttd6gb4bxf666vlt57r0000gp/T/jnilib-7676089916881629931.tmp: dlopen(/private/var/folders/rm/_bdtrttd6gb4bxf666vlt57r0000gp/T/jnilib-7676089916881629931.tmp, 0x0001): tried: '/private/var/folders/rm/_bdtrttd6gb4bxf666vlt57r0000gp/T/jnilib-7676089916881629931.tmp' (mach-o file, but is an incompatible architecture (have (x86_64), need (arm64e)))
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1950)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1832)
at java.lang.Runtime.load0(Runtime.java:811)
at java.lang.System.load(System.java:1088)
at org.apache.arrow.dataset.jni.JniLoader.load(JniLoader.java:88)
at org.apache.arrow.dataset.jni.JniLoader.loadRemaining(JniLoader.java:73)
at org.apache.arrow.dataset.jni.JniLoader.ensureLoaded(JniLoader.java:60)
at org.apache.arrow.dataset.jni.NativeMemoryPool.getDefault(NativeMemoryPool.java:34)
at demo.ArrowParquetReader$.delayedEndpoint$demo$ArrowParquetReader$1(ArrowParquetReader.scala:23)
at demo.ArrowParquetReader$delayedInit$body.apply(ArrowParquetReader.scala:14)
at scala.Function0.apply$mcV$sp(Function0.scala:39)
at scala.Function0.apply$mcV$sp$(Function0.scala:39)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
at scala.App.$anonfun$main$1(App.scala:76)
at scala.App.$anonfun$main$1$adapted(App.scala:76)
at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
at scala.collection.AbstractIterable.foreach(Iterable.scala:926)

```

Contributor guide

Open the contributing guide

Research direction

Start with the stack-trace entry points in ArrowParquetReader.scala:23, NativeMemoryPool.getDefault, and org.apache.arrow.dataset.jni.JniLoader.load. Check how the Arrow 9.0.0 dependencies select the native library for the reported arm64e environment; done means the Parquet reader starts without the UnsatisfiedLinkError.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.