apache / apache/arrow-java

[Java] Can't initialize RootAllocator when Unsafe isn't present even when using the NettyAllocator

未关闭
#123 15 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Java
星标
94
派生
152
平均合并
3 天 16 小时
30 天内合并 PR
11

描述

### Describe the bug, including details regarding any error messages, version, and platform.

In an Alpine docker container, the JVM doesn't have the Unsafe jars (for version 17 of openjdk). Even though we set the properties to use the Netty allocator instead, Arrow tries to load the Unsafe jars anyway, doesn't catch the exception and throws it to the caller of new RootAllocator().

The problem is this line of code:
at org.apache.arrow.memory.NettyAllocationManager.(NettyAllocationManager.java:51)
` private static final PooledByteBufAllocatorL INNER_ALLOCATOR = new PooledByteBufAllocatorL();`

It is only used a couple of times and only for default initialization. Perhaps just hard-code those defaults instead or just pull them from the same properties Netty does. You have to be really careful with static variables and if they throw exceptions, they probably aren't suitable for being in a static variable.

Here is the exception and relevant output:

Exception:
```
java.lang.UnsupportedOperationException: sun.misc.Unsafe unavailable
at io.netty.util.internal.CleanerJava9.(CleanerJava9.java:68)
at io.netty.util.internal.PlatformDependent.(PlatformDependent.java:191)
at io.netty.buffer.PooledByteBufAllocator.(PooledByteBufAllocator.java:116)
at io.netty.buffer.PooledByteBufAllocatorL.(PooledByteBufAllocatorL.java:49)
at org.apache.arrow.memory.NettyAllocationManager.(NettyAllocationManager.java:51)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:375)
at org.apache.arrow.memory.DefaultAllocationManagerOption.getFactory(DefaultAllocationManagerOption.java:108)
at org.apache.arrow.memory.DefaultAllocationManagerOption.getNettyFactory(DefaultAllocationManagerOption.java:127)
at org.apache.arrow.memory.DefaultAllocationManagerOption.getDefaultAllocationManagerFactory(DefaultAllocationManagerOption.java:91)
at org.apache.arrow.memory.BaseAllocator$Config.getAllocationManagerFactory(BaseAllocator.java:773)
at org.apache.arrow.memory.ImmutableConfig.access$801(ImmutableConfig.java:24)
at org.apache.arrow.memory.ImmutableConfig$InitShim.getAllocationManagerFactory(ImmutableConfig.java:83)
at org.apache.arrow.memory.ImmutableConfig.(ImmutableConfig.java:47)
at org.apache.arrow.memory.ImmutableConfig.(ImmutableConfig.java:24)
at org.apache.arrow.memory.ImmutableConfig$Builder.build(ImmutableConfig.java:485)
at org.apache.arrow.memory.BaseAllocator.(BaseAllocator.java:62)
at schema:public//org.seme.hadoop.ArrowFileWriterUDF.writeArrowFile(ArrowFileWriterUDF.java:131)
at org.postgresql.pljava.internal@1.6.5/org.postgresql.pljava.internal.EntryPoints.lambda$invocable$0(EntryPoints.java:130)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
at org.postgresql.pljava.internal@1.6.5/org.postgresql.pljava.internal.EntryPoints.doPrivilegedAndUnwrap(EntryPoints.java:312)
at org.postgresql.pljava.internal@1.6.5/org.postgresql.pljava.internal.EntryPoints.invoke(EntryPoints.java:158)
```

Debug output:
```
18:37:28.295 [main] DEBUG io.netty.util.ResourceLeakDetector -- -Dio.netty.leakDetection.level: simple
18:37:28.295 [main] DEBUG io.netty.util.ResourceLeakDetector -- -Dio.netty.leakDetection.targetRecords: 4
18:37:28.305 [main] DEBUG io.netty.util.internal.PlatformDependent0 -- -Dio.netty.noUnsafe: true
18:37:28.305 [main] DEBUG io.netty.util.internal.PlatformDependent0 -- sun.misc.Unsafe: unavailable (io.netty.noUnsafe)
18:37:28.305 [main] DEBUG io.netty.util.internal.PlatformDependent0 -- Java version: 17
18:37:28.306 [main] DEBUG io.netty.util.internal.PlatformDependent0 -- java.nio.DirectByteBuffer.(long, {int,long}): unavailable
18:37:28.307 [main] DEBUG io.netty.util.internal.PlatformDependent -- maxDirectMemory: 16802381824 bytes (maybe)
18:37:28.307 [main] DEBUG io.netty.util.internal.PlatformDependent -- -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
18:37:28.307 [main] DEBUG io.netty.util.internal.PlatformDependent -- -Dio.netty.bitMode: 64 (sun.arch.data.model)
18:37:28.307 [main] DEBUG io.netty.util.internal.PlatformDependent -- -Dio.netty.maxDirectMemory: -1 bytes
18:37:28.307 [main] DEBUG io.netty.util.internal.PlatformDependent -- -Dio.netty.uninitializedArrayAllocationThreshold: -1
18:37:28.309 [main] DEBUG io.netty.util.internal.PlatformDependent -- -Dio.netty.noPreferDirect: true
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.numHeapArenas: 16
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.numDirectArenas: 16
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.pageSize: 8192
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.maxOrder: 9
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.chunkSize: 4194304
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.smallCacheSize: 256
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.normalCacheSize: 64
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.maxCachedBufferCapacity: 32768
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.cacheTrimInterval: 8192
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.cacheTrimIntervalMillis: 0
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.useCacheForAllThreads: false
18:37:28.310 [main] DEBUG io.netty.buffer.PooledByteBufAllocator -- -Dio.netty.allocator.maxCachedByteBuffersPerChunk: 1023
18:37:28.313 [main] DEBUG io.netty.util.internal.InternalThreadLocalMap -- -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
18:37:28.313 [main] DEBUG io.netty.util.internal.InternalThreadLocalMap -- -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
18:37:28.320 [main] DEBUG io.netty.buffer.AbstractByteBuf -- -Dio.netty.buffer.checkAccessible: true
18:37:28.320 [main] DEBUG io.netty.buffer.AbstractByteBuf -- -Dio.netty.buffer.checkBounds: true
18:37:28.321 [main] DEBUG io.netty.util.ResourceLeakDetectorFactory -- Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@7cc5bdbd
18:37:28.400 [shutdown-hook-0] DEBUG org.apache.hadoop.hdfs.DataStreamer -- block==null waiting for ack for: -1
```

### Component(s)

Java

贡献指南

打开贡献指南

调研方向

从 PooledByteBufAllocatorL 的静态初始化处开始查看 NettyAllocationManager.java,然后跟踪 DefaultAllocationManagerOption.java 和新的入口点 new RootAllocator()。在 Alpine 容器中使用 Java 17 重现该故障,并验证在 Unsafe 不可用时选择 Netty 分配器不再失败。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
backend
Issue 类型
缺陷
难度
3/5
预计耗时
1-2 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
45/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。