[Bug] AppendOnly Table use managed memory write throws error:java.lang.IllegalArgumentException: The fraction of memory to allocate should not be 0. Please make sure that all types of managed memory consumers contained in the job are configured with a non-negative weight via `taskmanager.memory.managed.consumer-weights`
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
paimon-flink-1.18-1.0-20240923.002611-17.jar
### Compute Engine
flink1.18
### Minimal reproduce step
set 'table.exec.sink.upsert-materialize'='NONE';
set 'state.backend.type'='rocksdb';
CREATE CATALOG paimon_catalog WITH (
'type'='paimon',
'warehouse'='hdfs:///paimon/warehouse'
);
USE CATALOG paimon_catalog;
CREATE TABLE if NOT EXISTS
`default`.`ods_app_event_data_detail_dt_test` (
`id` string,
`app_key` string COMMENT 'app唯一标识,新接入app时,需申请。',
`dt` string COMMENT '分区日期天,格式化取自send_time'
) PARTITIONED BY (dt)
WITH
(
'sink.use-managed-memory-allocator' = 'true'
);
insert into `default`.`ods_app_event_data_detail_dt_test` values('1','2','2024-09-20');
then exception throws:
java.lang.IllegalArgumentException: The fraction of memory to allocate should not be 0. Please make sure that all types of managed memory consumers contained in the job are configured with a non-negative weight via `taskmanager.memory.managed.consumer-weights`.
at org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:160) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.memory.MemoryManager.validateFraction(MemoryManager.java:672) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.memory.MemoryManager.computeMemorySize(MemoryManager.java:653) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.paimon.flink.utils.ManagedMemoryUtils.computeManagedMemory(ManagedMemoryUtils.java:42) ~[paimon-flink-1.18-1.0-20240923.002611-17.jar:1.0-SNAPSHOT]
at org.apache.paimon.flink.sink.PrepareCommitOperator.setup(PrepareCommitOperator.java:71) ~[paimon-flink-1.18-1.0-20240923.002611-17.jar:1.0-SNAPSHOT]
at org.apache.flink.streaming.api.operators.SimpleOperatorFactory.createStreamOperator(SimpleOperatorFactory.java:84) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:81) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:214) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.(RegularOperatorChain.java:60) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:707) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:693) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:953) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:922) [flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:746) [flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) [flink-dist-1.18.0.jar:1.18.0]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
### What doesn't meet your expectations?
error
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with paimon-flink's ManagedMemoryUtils.computeManagedMemory and PrepareCommitOperator.setup, which appear in the stack trace. Run the provided Flink 1.18 SQL reproduction with sink.use-managed-memory-allocator enabled, then verify that the insert no longer throws the zero-memory-fraction exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100