apache / apache/paimon

[Bug] AppendOnly Table use managed memory write throws error:java.lang.IllegalArgumentException: The fraction of memory to allocate should not be 0. Please make sure that all types of managed memory consumers contained in the job are configured with a non-negative weight via `taskmanager.memory.managed.consumer-weights`

Open
#4,232 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

paimon-flink-1.18-1.0-20240923.002611-17.jar

### Compute Engine

flink1.18

### Minimal reproduce step

set 'table.exec.sink.upsert-materialize'='NONE';
set 'state.backend.type'='rocksdb';

CREATE CATALOG paimon_catalog WITH (
'type'='paimon',
'warehouse'='hdfs:///paimon/warehouse'
);

USE CATALOG paimon_catalog;

CREATE TABLE if NOT EXISTS
`default`.`ods_app_event_data_detail_dt_test` (
`id` string,
`app_key` string COMMENT 'app唯一标识,新接入app时,需申请。',
`dt` string COMMENT '分区日期天,格式化取自send_time'
) PARTITIONED BY (dt)
WITH
(
'sink.use-managed-memory-allocator' = 'true'
);

insert into `default`.`ods_app_event_data_detail_dt_test` values('1','2','2024-09-20');

then exception throws:
java.lang.IllegalArgumentException: The fraction of memory to allocate should not be 0. Please make sure that all types of managed memory consumers contained in the job are configured with a non-negative weight via `taskmanager.memory.managed.consumer-weights`.
at org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:160) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.memory.MemoryManager.validateFraction(MemoryManager.java:672) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.memory.MemoryManager.computeMemorySize(MemoryManager.java:653) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.paimon.flink.utils.ManagedMemoryUtils.computeManagedMemory(ManagedMemoryUtils.java:42) ~[paimon-flink-1.18-1.0-20240923.002611-17.jar:1.0-SNAPSHOT]
at org.apache.paimon.flink.sink.PrepareCommitOperator.setup(PrepareCommitOperator.java:71) ~[paimon-flink-1.18-1.0-20240923.002611-17.jar:1.0-SNAPSHOT]
at org.apache.flink.streaming.api.operators.SimpleOperatorFactory.createStreamOperator(SimpleOperatorFactory.java:84) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:81) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:214) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.(RegularOperatorChain.java:60) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:707) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:693) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:953) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:922) [flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:746) [flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) [flink-dist-1.18.0.jar:1.18.0]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]

### What doesn't meet your expectations?

error

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with paimon-flink's ManagedMemoryUtils.computeManagedMemory and PrepareCommitOperator.setup, which appear in the stack trace. Run the provided Flink 1.18 SQL reproduction with sink.use-managed-memory-allocator enabled, then verify that the insert no longer throws the zero-memory-fraction exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.