NVIDIA / NVIDIA/cudf

[BUG][JNI] Research reasons for the JNI pinned pool becoming very slow

Open
#12,341 0 comments 1 reaction 0 assignees View on GitHub
bug Java Spark
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

I have seen in jobs that have a lot of host memory usage, cases where executors looked to be hanging while trying to satisfy an allocation from the pinned pool in the java side of things:

```
Thread 25389: (state = IN_JAVA)
- java.util.TreeMap.successor(java.util.TreeMap$Entry) @bci=30, line=2154 (Compiled frame; information may be imprecise)
- java.util.TreeMap$KeySpliterator.tryAdvance(java.util.function.Consumer) @bci=45, line=2769 (Compiled frame)
- java.util.stream.ReferencePipeline.forEachWithCancel(java.util.Spliterator, java.util.stream.Sink) @bci=11, line=126 (Compiled frame)
- java.util.stream.AbstractPipeline.copyIntoWithCancel(java.util.stream.Sink, java.util.Spliterator) @bci=32, line=499 (Compiled frame)
- java.util.stream.AbstractPipeline.copyInto(java.util.stream.Sink, java.util.Spliterator) @bci=49, line=486 (Compiled frame)
- java.util.stream.AbstractPipeline.wrapAndCopyInto(java.util.stream.Sink, java.util.Spliterator) @bci=13, line=472 (Compiled frame)
- java.util.stream.FindOps$FindOp.evaluateSequential(java.util.stream.PipelineHelper, java.util.Spliterator) @bci=14, line=152 (Compiled frame)
- java.util.stream.AbstractPipeline.evaluate(java.util.stream.TerminalOp) @bci=88, line=234 (Compiled frame)
- java.util.stream.ReferencePipeline.findFirst() @bci=5, line=531 (Compiled frame)
- ai.rapids.cudf.PinnedMemoryPool.tryAllocateInternal(long) @bci=60, line=283 (Compiled frame)
- ai.rapids.cudf.PinnedMemoryPool.tryAllocate(long) @bci=12, line=224 (Compiled frame)
```

```
Thread 22565: (state = IN_JAVA)
- java.util.stream.ReferencePipeline.forEachWithCancel(java.util.Spliterator, java.util.stream.Sink) @bci=16, line=126 (Compiled frame; information may be imprecise)
- java.util.stream.AbstractPipeline.copyIntoWithCancel(java.util.stream.Sink, java.util.Spliterator) @bci=32, line=499 (Compiled frame)
- java.util.stream.AbstractPipeline.copyInto(java.util.stream.Sink, java.util.Spliterator) @bci=49, line=486 (Compiled frame)
- java.util.stream.AbstractPipeline.wrapAndCopyInto(java.util.stream.Sink, java.util.Spliterator) @bci=13, line=472 (Compiled frame)
- java.util.stream.FindOps$FindOp.evaluateSequential(java.util.stream.PipelineHelper, java.util.Spliterator) @bci=14, line=152 (Compiled frame)
- java.util.stream.AbstractPipeline.evaluate(java.util.stream.TerminalOp) @bci=88, line=234 (Compiled frame)
- java.util.stream.ReferencePipeline.findFirst() @bci=5, line=531 (Compiled frame)
- ai.rapids.cudf.PinnedMemoryPool.tryAllocateInternal(long) @bci=60, line=283 (Compiled frame)
- ai.rapids.cudf.PinnedMemoryPool.tryAllocate(long) @bci=12, line=224 (Compiled frame)

```

I don't have a repro case yet and that's part of this task. I have seen this happen after the spark-plugin has spilled a lot of memory to the host, and it seems we should have exhausted this pool. Theories could go from this pool structure being somehow corrupted, or there being an insane number of blocks in this pool that it is just taking a lot of time for it to find free blocks.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.