apache / apache/arrow-java

[Java] Can't initialize RootAllocator when Unsafe isn't present even when using the NettyAllocator

Aperta
#123 15 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Java
Stelle
94
Fork
152
Merge medio
3g 16h
PR unite (30g)
11

Descrizione

### Describe the bug, including details regarding any error messages, version, and platform.

In an Alpine docker container, the JVM doesn't have the Unsafe jars (for version 17 of openjdk). Even though we set the properties to use the Netty allocator instead, Arrow tries to load the Unsafe jars anyway, doesn't catch the exception and throws it to the caller of new RootAllocator().

The problem is this line of code:
at org.apache.arrow.memory.NettyAllocationManager.(NettyAllocationManager.java:51)
` private static final PooledByteBufAllocatorL INNER_ALLOCATOR = new PooledByteBufAllocatorL();`

It is only used a couple of times and only for default initialization. Perhaps just hard-code those defaults instead or just pull them from the same properties Netty does. You have to be really careful with static variables and if they throw exceptions, they probably aren't suitable for being in a static variable.

Here is the exception and relevant output:

Exception:
```
java.lang.UnsupportedOperationException: sun.misc.Unsafe unavailable
at io.netty.util.internal.CleanerJava9.(CleanerJava9.java:68)
at io.netty.util.internal.PlatformDependent.(PlatformDependent.java:191)
at io.netty.buffer.PooledByteBufAllocator.(PooledByteBufAllocator.java:116)
at io.netty.buffer.PooledByteBufAllocatorL.(PooledByteBufAllocatorL.java:49)
at org.apache.arrow.memory.NettyAllocationManager.(NettyAllocationManager.java:51)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:375)
at org.apache.arrow.memory.DefaultAllocationManagerOption.getFactory(DefaultAllocationManagerOption.java:108)
at org.apache.arrow.memory.DefaultAllocationManagerOption.getNettyFactory(DefaultAllocationManagerOption.java:127)
at org.apache.arrow.memory.DefaultAllocationManagerOption.getDefaultAllocationManagerFactory(DefaultAllocationManagerOption.java:91)
at org.apache.arrow.memory.BaseAllocator$Config.getAllocationManagerFactory(BaseAllocator.java:773)
at org.apache.arrow.memory.ImmutableConfig.access$801(ImmutableConfig.java:24)
at org.apache.arrow.memory.ImmutableConfig$InitShim.getAllocationManagerFactory(ImmutableConfig.java:83)
at org.apache.arrow.memory.ImmutableConfig.(ImmutableConfig.java:47)
at org.apache.arrow.memory.ImmutableConfig.(ImmutableConfig.java:24)
at org.apache.arrow.memory.ImmutableConfig$Builder.build(ImmutableConfig.java:485)
at org.apache.arrow.memory.BaseAllocator.(BaseAllocator.java:62)
at schema:public//org.seme.hadoop.ArrowFileWriterUDF.writeArrowFile(ArrowFileWriterUDF.java:131)
at org.postgresql.pljava.internal@1.6.5/org.postgresql.pljava.internal.EntryPoints.lambda$invocable$0(EntryPoints.java:130)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
at org.postgresql.pljava.internal@1.6.5/org.postgresql.pljava.internal.EntryPoints.doPrivilegedAndUnwrap(EntryPoints.java:312)
at org.postgresql.pljava.internal@1.6.5/org.postgresql.pljava.internal.EntryPoints.invoke(EntryPoints.java:158)
```

Debug output:
```
18:37:28.295 [main] DEBUG io.netty.util.ResourceLeakDetector -- -Dio.netty.leakDetection.level: simple
18:37:28.295 [main] DEBUG io.netty.util.ResourceLeakDetector -- -Dio.netty.leakDetection.targetRecords: 4
18:37:28.305 [main] DEBUG io.netty.util.internal.PlatformDependent0 -- -Dio.netty.noUnsafe: true
18:37:28.305 [main] DEBUG io.netty.util.internal.PlatformDependent0 -- sun.misc.Unsafe: unavailable (io.netty.noUnsafe)
18:37:28.305 [main] DEBUG io.netty.util.internal.PlatformDependent0 -- Java version: 17
18:37:28.306 [main] DEBUG io.netty.util.internal.PlatformDependent0 -- java.nio.DirectByteBuffer.(long, {int,long}): unavailable
18:37:28.307 [main] DEBUG io.netty.util.internal.PlatformDependent -- maxDirectMemory: 16802381824 bytes (maybe)
18:37:28.307 [main] DEBUG io.netty.util.internal.PlatformDependent -- -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
18:37:28.307 [main] DEBUG io.netty.util.internal.PlatformDependent -- -Dio.netty.bitMode: 64 (sun.arch.data.model)
18:37:28.307 [main] DEBUG io.netty.util.internal.PlatformDependent -- -Dio.netty.maxDirectMemory: -1 bytes
18:37:28.307 [main] DEBUG io.netty.util.internal.PlatformDependent -- -Dio.netty.uninitializedArrayAllocationThreshold: -1
18:37:28.309 [main] DEBUG io.netty.util.internal.PlatformDependent -- -Dio.netty.noPreferDirect: true
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.numHeapArenas: 16
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.numDirectArenas: 16
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.pageSize: 8192
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.maxOrder: 9
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.chunkSize: 4194304
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.smallCacheSize: 256
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.normalCacheSize: 64
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.maxCachedBufferCapacity: 32768
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.cacheTrimInterval: 8192
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.cacheTrimIntervalMillis: 0
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.useCacheForAllThreads: false
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.maxCachedByteBuffersPerChunk: 1023
18:37:28.313 [main] DEBUG io.netty.util.internal.InternalThreadLocalMap -- -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
18:37:28.313 [main] DEBUG io.netty.util.internal.InternalThreadLocalMap -- -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
18:37:28.320 [main] DEBUG io.netty.buffer.AbstractByteBuf -- -Dio.netty.buffer.checkAccessible: true
18:37:28.320 [main] DEBUG io.netty.buffer.AbstractByteBuf -- -Dio.netty.buffer.checkBounds: true
18:37:28.321 [main] DEBUG io.netty.util.ResourceLeakDetectorFactory -- Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@7cc5bdbd
18:37:28.400 [shutdown-hook-0] DEBUG org.apache.hadoop.hdfs.DataStreamer -- block==null waiting for ack for: -1
```

### Component(s)

Java

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia da NettyAllocationManager.java durante l’inizializzazione statica di PooledByteBufAllocatorL, quindi segui DefaultAllocationManagerOption.java e il nuovo punto di ingresso new RootAllocator(). Riproduci il fallimento con Java 17 in un container Alpine e verifica che la selezione dell’allocatore Netty non fallisca più quando Unsafe non è disponibile.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
java
Ambito
backend
Tipo di issue
Bug
Difficoltà
3/5
Tempo stimato
1-2 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
45/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.