apache / apache/paimon

[Bug] AppendOnly Table use managed memory write throws error:java.lang.IllegalArgumentException: The fraction of memory to allocate should not be 0. Please make sure that all types of managed memory consumers contained in the job are configured with a non-negative weight via `taskmanager.memory.managed.consumer-weights`

Aperta
#4,232 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
bug
Lingua principale
Java
Stelle
3.4k
Fork
1.4k
Merge medio
1g 9h
PR unite (30g)
423

Descrizione

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

paimon-flink-1.18-1.0-20240923.002611-17.jar

### Compute Engine

flink1.18

### Minimal reproduce step

set 'table.exec.sink.upsert-materialize'='NONE';
set 'state.backend.type'='rocksdb';

CREATE CATALOG paimon_catalog WITH (
'type'='paimon',
'warehouse'='hdfs:///paimon/warehouse'
);

USE CATALOG paimon_catalog;

CREATE TABLE if NOT EXISTS
`default`.`ods_app_event_data_detail_dt_test` (
`id` string,
`app_key` string COMMENT 'app唯一标识,新接入app时,需申请。',
`dt` string COMMENT '分区日期天,格式化取自send_time'
) PARTITIONED BY (dt)
WITH
(
'sink.use-managed-memory-allocator' = 'true'
);

insert into `default`.`ods_app_event_data_detail_dt_test` values('1','2','2024-09-20');

then exception throws:
java.lang.IllegalArgumentException: The fraction of memory to allocate should not be 0. Please make sure that all types of managed memory consumers contained in the job are configured with a non-negative weight via `taskmanager.memory.managed.consumer-weights`.
at org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:160) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.memory.MemoryManager.validateFraction(MemoryManager.java:672) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.memory.MemoryManager.computeMemorySize(MemoryManager.java:653) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.paimon.flink.utils.ManagedMemoryUtils.computeManagedMemory(ManagedMemoryUtils.java:42) ~[paimon-flink-1.18-1.0-20240923.002611-17.jar:1.0-SNAPSHOT]
at org.apache.paimon.flink.sink.PrepareCommitOperator.setup(PrepareCommitOperator.java:71) ~[paimon-flink-1.18-1.0-20240923.002611-17.jar:1.0-SNAPSHOT]
at org.apache.flink.streaming.api.operators.SimpleOperatorFactory.createStreamOperator(SimpleOperatorFactory.java:84) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:81) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:214) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.(RegularOperatorChain.java:60) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:707) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:693) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:953) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:922) [flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:746) [flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) [flink-dist-1.18.0.jar:1.18.0]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]

### What doesn't meet your expectations?

error

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Inizia con ManagedMemoryUtils.computeManagedMemory e PrepareCommitOperator.setup di paimon-flink, che compaiono nello stack trace. Esegui la riproduzione SQL di Flink 1.18 fornita con sink.use-managed-memory-allocator abilitato, quindi verifica che l'insert non generi più la zero-memory-fraction exception.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
java
Ambito
data-engineering
Tipo di issue
Bug
Difficoltà
3/5
Tempo stimato
1-2 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.