apache / apache/hudi

[SUPPORT] Flink write meet NoSuchElementException: FileID does not exist

Open
#7,470 10 comments 0 reactions 0 assignees View on GitHub
area:index area:writer engine:flink priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

Flink write hudi, upsert, state index

after running few hours, meet error: Caused by: java.util.NoSuchElementException: FileID 7a41bae9-0ee9-4415-acff-974ad33d943c of partition path region=CN/date=2022-12-15/subver=ANDROID does not exist.

**To Reproduce**

Steps to reproduce the behavior:

1.flink job upsert hudi table with state index.
2.wait for job running for 2-6 hours.
3.the job will encounter the exception desc above.
4.The checkpoint will fail and the job can never complete new checkpoint any more.

**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment Description**

* Hudi version : 0.12.1

**Additional context**

flink table options:
```
'table.type' = 'COPY_ON_WRITE',
'hoodie.datasource.write.recordkey.field' = 'pk_field',
'hoodie.datasource.write.hive_style_partitioning' = 'true',

-- write config
'write.operation' = 'upsert',
'hoodie.parquet.small.file.limit' = '104857600',
'hoodie.parquet.max.file.size' = '268435456',
'hoodie.cleaner.policy' = 'KEEP_LATEST_BY_HOURS',
'hoodie.cleaner.hours.retained' = '72',
'hoodie.datasource.write.keygenerator.class' = 'org.apache.hudi.keygen.ComplexAvroKeyGenerator',
'write.precombine' = 'true',
'write.precombine.field' = 'ts',
'index.state.ttl' = '60',
'index.type' = 'FLINK_STATE',
'index.bootstrap.enabled' = 'false',

-- tasks parallelism
'write.bucket_assign.tasks' = '256',
'write.tasks' = '256',

-- sync metadata to hive
'hive_sync.partition_extractor_class' = 'org.apache.hudi.hive.MultiPartKeysValueExtractor',
'hive_sync.enable' = 'true',
'hive_sync.db' = 'my_db',
'hive_sync.table' = 'my_table',
'hive_sync.metastore.uris'='thrift://...',
'hive_sync.use_jdbc'= 'false',
'hive_sync.mode'='hms',
```

**Stacktrace**
```
Caused by: java.util.NoSuchElementException: FileID 7a41bae9-0ee9-4415-acff-974ad33d943c of partition path region=CN/date=2022-12-15/subver=ANDROID does not exist.
at org.apache.hudi.io.HoodieMergeHandle.getLatestBaseFile(HoodieMergeHandle.java:155)
at org.apache.hudi.io.HoodieMergeHandle.(HoodieMergeHandle.java:121)
at org.apache.hudi.io.FlinkMergeHandle.(FlinkMergeHandle.java:70)
at org.apache.hudi.client.HoodieFlinkWriteClient.getOrCreateWriteHandle(HoodieFlinkWriteClient.java:600)
at org.apache.hudi.client.HoodieFlinkWriteClient.upsert(HoodieFlinkWriteClient.java:154)
at org.apache.hudi.sink.StreamWriteFunction.lambda$initWriteFunction$1(StreamWriteFunction.java:185)
at org.apache.hudi.sink.StreamWriteFunction.lambda$flushRemaining$7(StreamWriteFunction.java:463)
at java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608)
at org.apache.hudi.sink.StreamWriteFunction.flushRemaining(StreamWriteFunction.java:455)
at org.apache.hudi.sink.StreamWriteFunction.snapshotState(StreamWriteFunction.java:131)
at org.apache.hudi.sink.common.AbstractStreamWriteFunction.snapshotState(AbstractStreamWriteFunction.java:168)
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.trySnapshotFunctionState(StreamingFunctionUtils.java:118)
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.snapshotFunctionState(StreamingFunctionUtils.java:99)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.snapshotState(AbstractUdfStreamOperator.java:87)
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:222)
... 27 more
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with org.apache.hudi.io.HoodieMergeHandle.java:155 and org.apache.hudi.io.FlinkMergeHandle.java:70, then trace the call from HoodieFlinkWriteClient.upsert shown in the stack trace. Reproduce the failure using the reported Hudi 0.12.1 Flink upsert configuration with the state index and cleaner settings. Done means the missing FileID cause is identified and checkpoint processing no longer fails or the limitation is clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.