apache / apache/paimon

[Bug] NPE after using flink manage memory

Open
#1,332 14 comments 0 reactions 1 assignee Claimed by @q977734161 View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Paimon version

```
2023-06-08 12:47:49
java.io.IOException: java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.paimon.flink.sink.RowDataStoreWriteOperator.processElement(RowDataStoreWriteOperator.java:161)
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:237)
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:146)
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:110)
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:550)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:839)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:788)
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:931)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.paimon.memory.MemoryPoolFactory.preemptMemory(MemoryPoolFactory.java:63)
at org.apache.paimon.memory.MemoryPoolFactory.access$200(MemoryPoolFactory.java:27)
at org.apache.paimon.memory.MemoryPoolFactory$OwnerMemoryPool.nextSegment(MemoryPoolFactory.java:98)
at org.apache.paimon.data.SimpleCollectingOutputView.nextSegment(SimpleCollectingOutputView.java:69)
at org.apache.paimon.data.AbstractPagedOutputView.advance(AbstractPagedOutputView.java:122)
at org.apache.paimon.data.SimpleCollectingOutputView.reset(SimpleCollectingOutputView.java:59)
at org.apache.paimon.sort.BinaryInMemorySortBuffer.tryInitialize(BinaryInMemorySortBuffer.java:120)
at org.apache.paimon.sort.BinaryInMemorySortBuffer.write(BinaryInMemorySortBuffer.java:166)
at org.apache.paimon.mergetree.SortBufferWriteBuffer.put(SortBufferWriteBuffer.java:106)
at org.apache.paimon.mergetree.MergeTreeWriter.write(MergeTreeWriter.java:146)
at org.apache.paimon.mergetree.MergeTreeWriter.write(MergeTreeWriter.java:53)
at org.apache.paimon.operation.AbstractFileStoreWrite.write(AbstractFileStoreWrite.java:100)
at org.apache.paimon.table.sink.TableWriteImpl.writeAndReturn(TableWriteImpl.java:101)
at org.apache.paimon.flink.sink.StoreSinkWriteImpl.write(StoreSinkWriteImpl.java:91)
at org.apache.paimon.flink.sink.GlobalFullCompactionSinkWrite.write(GlobalFullCompactionSinkWrite.java:100)
at org.apache.paimon.flink.sink.RowDataStoreWriteOperator.processElement(RowDataStoreWriteOperator.java:159)
... 13 more
Caused by: java.lang.NullPointerException
at java.lang.System.arraycopy(Native Method)
at org.apache.hadoop.io.Text.set(Text.java:225)
at org.apache.paimon.shade.org.apache.orc.impl.StringRedBlackTree.add(StringRedBlackTree.java:59)
at org.apache.paimon.shade.org.apache.orc.impl.writer.StringTreeWriter.writeBatch(StringTreeWriter.java:70)
at org.apache.paimon.shade.org.apache.orc.impl.writer.StructTreeWriter.writeRootBatch(StructTreeWriter.java:56)
at org.apache.paimon.shade.org.apache.orc.impl.WriterImpl.addRowBatch(WriterImpl.java:557)
at org.apache.paimon.format.orc.writer.OrcBulkWriter.flush(OrcBulkWriter.java:60)
at org.apache.paimon.io.SingleFileWriter.close(SingleFileWriter.java:143)
at org.apache.paimon.io.RollingFileWriter.closeCurrentWriter(RollingFileWriter.java:111)
at org.apache.paimon.io.RollingFileWriter.close(RollingFileWriter.java:151)
at org.apache.paimon.mergetree.MergeTreeWriter.flushWriteBuffer(MergeTreeWriter.java:208)
at org.apache.paimon.mergetree.MergeTreeWriter.flushMemory(MergeTreeWriter.java:180)
at org.apache.paimon.memory.MemoryPoolFactory.preemptMemory(MemoryPoolFactory.java:61)
```

### Compute Engine

flink v1.17.0

### Minimal reproduce step
0. An old task encountered an oom,I plan to use manage memory to optimize this situation.
1. set manage memory 3.5GB
2. use flink manage memory /*+ OPTIONS('sink.use-managed-memory-allocator'='true', 'sink.managed.writer-buffer-memory'='256M') */

### What doesn't meet your expectations?

NPE

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.