[Bug] AppendOnly Table use managed memory write throws error:java.lang.IllegalArgumentException: The fraction of memory to allocate should not be 0. Please make sure that all types of managed memory consumers contained in the job are configured with a non-negative weight via `taskmanager.memory.managed.consumer-weights`
- Lingua principale
- Java
- Stelle
- 3.4k
- Fork
- 1.4k
- Merge medio
- 1g 9h
- PR unite (30g)
- 423
Descrizione
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
paimon-flink-1.18-1.0-20240923.002611-17.jar
### Compute Engine
flink1.18
### Minimal reproduce step
set 'table.exec.sink.upsert-materialize'='NONE';
set 'state.backend.type'='rocksdb';
CREATE CATALOG paimon_catalog WITH (
'type'='paimon',
'warehouse'='hdfs:///paimon/warehouse'
);
USE CATALOG paimon_catalog;
CREATE TABLE if NOT EXISTS
`default`.`ods_app_event_data_detail_dt_test` (
`id` string,
`app_key` string COMMENT 'app唯一标识,新接入app时,需申请。',
`dt` string COMMENT '分区日期天,格式化取自send_time'
) PARTITIONED BY (dt)
WITH
(
'sink.use-managed-memory-allocator' = 'true'
);
insert into `default`.`ods_app_event_data_detail_dt_test` values('1','2','2024-09-20');
then exception throws:
java.lang.IllegalArgumentException: The fraction of memory to allocate should not be 0. Please make sure that all types of managed memory consumers contained in the job are configured with a non-negative weight via `taskmanager.memory.managed.consumer-weights`.
at org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:160) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.memory.MemoryManager.validateFraction(MemoryManager.java:672) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.memory.MemoryManager.computeMemorySize(MemoryManager.java:653) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.paimon.flink.utils.ManagedMemoryUtils.computeManagedMemory(ManagedMemoryUtils.java:42) ~[paimon-flink-1.18-1.0-20240923.002611-17.jar:1.0-SNAPSHOT]
at org.apache.paimon.flink.sink.PrepareCommitOperator.setup(PrepareCommitOperator.java:71) ~[paimon-flink-1.18-1.0-20240923.002611-17.jar:1.0-SNAPSHOT]
at org.apache.flink.streaming.api.operators.SimpleOperatorFactory.createStreamOperator(SimpleOperatorFactory.java:84) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:81) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:214) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.(RegularOperatorChain.java:60) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:707) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:693) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:953) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:922) [flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:746) [flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) [flink-dist-1.18.0.jar:1.18.0]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
### What doesn't meet your expectations?
error
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Direzione di ricerca
Inizia con ManagedMemoryUtils.computeManagedMemory e PrepareCommitOperator.setup di paimon-flink, che compaiono nello stack trace. Esegui la riproduzione SQL di Flink 1.18 fornita con sink.use-managed-memory-allocator abilitato, quindi verifica che l'insert non generi più la zero-memory-fraction exception.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- java
- Ambito
- data-engineering
- Tipo di issue
- Bug
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Stato di attività
- Ferma
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 35/100