[SUPPORT] Flink write meet NoSuchElementException: FileID does not exist
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
Flink write hudi, upsert, state index
after running few hours, meet error: Caused by: java.util.NoSuchElementException: FileID 7a41bae9-0ee9-4415-acff-974ad33d943c of partition path region=CN/date=2022-12-15/subver=ANDROID does not exist.
**To Reproduce**
Steps to reproduce the behavior:
1.flink job upsert hudi table with state index.
2.wait for job running for 2-6 hours.
3.the job will encounter the exception desc above.
4.The checkpoint will fail and the job can never complete new checkpoint any more.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : 0.12.1
**Additional context**
flink table options:
```
'table.type' = 'COPY_ON_WRITE',
'hoodie.datasource.write.recordkey.field' = 'pk_field',
'hoodie.datasource.write.hive_style_partitioning' = 'true',
-- write config
'write.operation' = 'upsert',
'hoodie.parquet.small.file.limit' = '104857600',
'hoodie.parquet.max.file.size' = '268435456',
'hoodie.cleaner.policy' = 'KEEP_LATEST_BY_HOURS',
'hoodie.cleaner.hours.retained' = '72',
'hoodie.datasource.write.keygenerator.class' = 'org.apache.hudi.keygen.ComplexAvroKeyGenerator',
'write.precombine' = 'true',
'write.precombine.field' = 'ts',
'index.state.ttl' = '60',
'index.type' = 'FLINK_STATE',
'index.bootstrap.enabled' = 'false',
-- tasks parallelism
'write.bucket_assign.tasks' = '256',
'write.tasks' = '256',
-- sync metadata to hive
'hive_sync.partition_extractor_class' = 'org.apache.hudi.hive.MultiPartKeysValueExtractor',
'hive_sync.enable' = 'true',
'hive_sync.db' = 'my_db',
'hive_sync.table' = 'my_table',
'hive_sync.metastore.uris'='thrift://...',
'hive_sync.use_jdbc'= 'false',
'hive_sync.mode'='hms',
```
**Stacktrace**
```
Caused by: java.util.NoSuchElementException: FileID 7a41bae9-0ee9-4415-acff-974ad33d943c of partition path region=CN/date=2022-12-15/subver=ANDROID does not exist.
at org.apache.hudi.io.HoodieMergeHandle.getLatestBaseFile(HoodieMergeHandle.java:155)
at org.apache.hudi.io.HoodieMergeHandle.(HoodieMergeHandle.java:121)
at org.apache.hudi.io.FlinkMergeHandle.(FlinkMergeHandle.java:70)
at org.apache.hudi.client.HoodieFlinkWriteClient.getOrCreateWriteHandle(HoodieFlinkWriteClient.java:600)
at org.apache.hudi.client.HoodieFlinkWriteClient.upsert(HoodieFlinkWriteClient.java:154)
at org.apache.hudi.sink.StreamWriteFunction.lambda$initWriteFunction$1(StreamWriteFunction.java:185)
at org.apache.hudi.sink.StreamWriteFunction.lambda$flushRemaining$7(StreamWriteFunction.java:463)
at java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608)
at org.apache.hudi.sink.StreamWriteFunction.flushRemaining(StreamWriteFunction.java:455)
at org.apache.hudi.sink.StreamWriteFunction.snapshotState(StreamWriteFunction.java:131)
at org.apache.hudi.sink.common.AbstractStreamWriteFunction.snapshotState(AbstractStreamWriteFunction.java:168)
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.trySnapshotFunctionState(StreamingFunctionUtils.java:118)
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.snapshotFunctionState(StreamingFunctionUtils.java:99)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.snapshotState(AbstractUdfStreamOperator.java:87)
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:222)
... 27 more
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with org.apache.hudi.io.HoodieMergeHandle.java:155 and org.apache.hudi.io.FlinkMergeHandle.java:70, then trace the call from HoodieFlinkWriteClient.upsert shown in the stack trace. Reproduce the failure using the reported Hudi 0.12.1 Flink upsert configuration with the state index and cleaner settings. Done means the missing FileID cause is identified and checkpoint processing no longer fails or the limitation is clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100