apache / apache/arrow-java

[Java] Can't initialize RootAllocator when Unsafe isn't present even when using the NettyAllocator

Abierto
#123 15 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Java
Estrellas
94
Forks
152
Merge medio
3 d 16 h
PR fusionados (30 d)
11

Descripción

### Describe the bug, including details regarding any error messages, version, and platform.

In an Alpine docker container, the JVM doesn't have the Unsafe jars (for version 17 of openjdk). Even though we set the properties to use the Netty allocator instead, Arrow tries to load the Unsafe jars anyway, doesn't catch the exception and throws it to the caller of new RootAllocator().

The problem is this line of code:
at org.apache.arrow.memory.NettyAllocationManager.(NettyAllocationManager.java:51)
` private static final PooledByteBufAllocatorL INNER_ALLOCATOR = new PooledByteBufAllocatorL();`

It is only used a couple of times and only for default initialization. Perhaps just hard-code those defaults instead or just pull them from the same properties Netty does. You have to be really careful with static variables and if they throw exceptions, they probably aren't suitable for being in a static variable.

Here is the exception and relevant output:

Exception:
```
java.lang.UnsupportedOperationException: sun.misc.Unsafe unavailable
at io.netty.util.internal.CleanerJava9.(CleanerJava9.java:68)
at io.netty.util.internal.PlatformDependent.(PlatformDependent.java:191)
at io.netty.buffer.PooledByteBufAllocator.(PooledByteBufAllocator.java:116)
at io.netty.buffer.PooledByteBufAllocatorL.(PooledByteBufAllocatorL.java:49)
at org.apache.arrow.memory.NettyAllocationManager.(NettyAllocationManager.java:51)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:375)
at org.apache.arrow.memory.DefaultAllocationManagerOption.getFactory(DefaultAllocationManagerOption.java:108)
at org.apache.arrow.memory.DefaultAllocationManagerOption.getNettyFactory(DefaultAllocationManagerOption.java:127)
at org.apache.arrow.memory.DefaultAllocationManagerOption.getDefaultAllocationManagerFactory(DefaultAllocationManagerOption.java:91)
at org.apache.arrow.memory.BaseAllocator$Config.getAllocationManagerFactory(BaseAllocator.java:773)
at org.apache.arrow.memory.ImmutableConfig.access$801(ImmutableConfig.java:24)
at org.apache.arrow.memory.ImmutableConfig$InitShim.getAllocationManagerFactory(ImmutableConfig.java:83)
at org.apache.arrow.memory.ImmutableConfig.(ImmutableConfig.java:47)
at org.apache.arrow.memory.ImmutableConfig.(ImmutableConfig.java:24)
at org.apache.arrow.memory.ImmutableConfig$Builder.build(ImmutableConfig.java:485)
at org.apache.arrow.memory.BaseAllocator.(BaseAllocator.java:62)
at schema:public//org.seme.hadoop.ArrowFileWriterUDF.writeArrowFile(ArrowFileWriterUDF.java:131)
at org.postgresql.pljava.internal@1.6.5/org.postgresql.pljava.internal.EntryPoints.lambda$invocable$0(EntryPoints.java:130)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
at org.postgresql.pljava.internal@1.6.5/org.postgresql.pljava.internal.EntryPoints.doPrivilegedAndUnwrap(EntryPoints.java:312)
at org.postgresql.pljava.internal@1.6.5/org.postgresql.pljava.internal.EntryPoints.invoke(EntryPoints.java:158)
```

Debug output:
```
18:37:28.295 [main] DEBUG io.netty.util.ResourceLeakDetector -- -Dio.netty.leakDetection.level: simple
18:37:28.295 [main] DEBUG io.netty.util.ResourceLeakDetector -- -Dio.netty.leakDetection.targetRecords: 4
18:37:28.305 [main] DEBUG io.netty.util.internal.PlatformDependent0 -- -Dio.netty.noUnsafe: true
18:37:28.305 [main] DEBUG io.netty.util.internal.PlatformDependent0 -- sun.misc.Unsafe: unavailable (io.netty.noUnsafe)
18:37:28.305 [main] DEBUG io.netty.util.internal.PlatformDependent0 -- Java version: 17
18:37:28.306 [main] DEBUG io.netty.util.internal.PlatformDependent0 -- java.nio.DirectByteBuffer.(long, {int,long}): unavailable
18:37:28.307 [main] DEBUG io.netty.util.internal.PlatformDependent -- maxDirectMemory: 16802381824 bytes (maybe)
18:37:28.307 [main] DEBUG io.netty.util.internal.PlatformDependent -- -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
18:37:28.307 [main] DEBUG io.netty.util.internal.PlatformDependent -- -Dio.netty.bitMode: 64 (sun.arch.data.model)
18:37:28.307 [main] DEBUG io.netty.util.internal.PlatformDependent -- -Dio.netty.maxDirectMemory: -1 bytes
18:37:28.307 [main] DEBUG io.netty.util.internal.PlatformDependent -- -Dio.netty.uninitializedArrayAllocationThreshold: -1
18:37:28.309 [main] DEBUG io.netty.util.internal.PlatformDependent -- -Dio.netty.noPreferDirect: true
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.numHeapArenas: 16
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.numDirectArenas: 16
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.pageSize: 8192
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.maxOrder: 9
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.chunkSize: 4194304
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.smallCacheSize: 256
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.normalCacheSize: 64
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.maxCachedBufferCapacity: 32768
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.cacheTrimInterval: 8192
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.cacheTrimIntervalMillis: 0
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.useCacheForAllThreads: false
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.maxCachedByteBuffersPerChunk: 1023
18:37:28.313 [main] DEBUG io.netty.util.internal.InternalThreadLocalMap -- -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
18:37:28.313 [main] DEBUG io.netty.util.internal.InternalThreadLocalMap -- -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
18:37:28.320 [main] DEBUG io.netty.buffer.AbstractByteBuf -- -Dio.netty.buffer.checkAccessible: true
18:37:28.320 [main] DEBUG io.netty.buffer.AbstractByteBuf -- -Dio.netty.buffer.checkBounds: true
18:37:28.321 [main] DEBUG io.netty.util.ResourceLeakDetectorFactory -- Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@7cc5bdbd
18:37:28.400 [shutdown-hook-0] DEBUG org.apache.hadoop.hdfs.DataStreamer -- block==null waiting for ack for: -1
```

### Component(s)

Java

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Comienza con NettyAllocationManager.java en la inicialización estática de PooledByteBufAllocatorL y, a continuación, sigue el recorrido por DefaultAllocationManagerOption.java y el nuevo punto de entrada new RootAllocator(). Reproduce el fallo con Java 17 en un contenedor Alpine y verifica que seleccionar el asignador de Netty ya no falle cuando Unsafe no esté disponible.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
java
Área
backend
Tipo de issue
Error
Dificultad
3/5
Tiempo estimado
1-2 días
Estado de actividad
Estancado
Claridad
Bastante claro
Aptitud para principiantes
45/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.