[SUPPORT] AWS Athena query fail when compaction is scheduled for MOR table
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
I'm using hudi 0.14.0 with flink 1.16.1 to store data from kafka to s3.
but Athena(Engine 3) query to MOR table is not working because of this error.
```
Error running query: HIVE_UNKNOWN_ERROR: io.trino.plugin.hive.s3.TrinoS3FileSystem$UnrecoverableS3OperationException: com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: ***; S3 Extended Request ID: ***; Proxy: null), S3 Extended Request ID: *** (Bucket: mybucket, Key: mytable/.hoodie/.aux/20231014095517882.compaction.requested)
```
This error occurs if compaction is scheduled.
After compaction is complete, query is working.
Here's flink hudi option (Java)
```
flinkHudiOptions.put(FlinkOptions.PATH.key(), basePath);
flinkHudiOptions.put(FlinkOptions.TABLE_TYPE.key(), HoodieTableType.MERGE_ON_READ.name());
flinkHudiOptions.put(FlinkOptions.OPERATION.key(), WriteOperationType.UPSERT.name());
flinkHudiOptions.put(FlinkOptions.PRECOMBINE_FIELD.key(), "event_time");
flinkHudiOptions.put(FlinkOptions.KEYGEN_CLASS_NAME.key(), "org.apache.hudi.keygen.ComplexKeyGenerator");
flinkHudiOptions.put(FlinkOptions.COMPACTION_ASYNC_ENABLED.key(), "true");
flinkHudiOptions.put(FlinkOptions.COMPACTION_TRIGGER_STRATEGY.key(), FlinkOptions.NUM_COMMITS);
flinkHudiOptions.put(FlinkOptions.COMPACTION_DELTA_COMMITS.key(), "5");
flinkHudiOptions.put(FlinkOptions.COMPACTION_MAX_MEMORY.key(), "1024");
flinkHudiOptions.put(FlinkOptions.METADATA_ENABLED.key(), "true");
flinkHudiOptions.put(HoodieMetadataConfig.ASYNC_INDEX_ENABLE.key(), "true");
flinkHudiOptions.put(HoodieMetadataConfig.ENABLE_METADATA_INDEX_COLUMN_STATS.key(), "true");
flinkHudiOptions.put(HoodieWriteConfig.WRITE_CONCURRENCY_MODE.key(), WriteConcurrencyMode.OPTIMISTIC_CONCURRENCY_CONTROL.name());
flinkHudiOptions.put(HoodieLockConfig.LOCK_PROVIDER_CLASS_NAME.key(), "org.apache.hudi.client.transaction.lock.InProcessLockProvider");
flinkHudiOptions.put(FlinkOptions.CLEAN_ASYNC_ENABLED.key(), "true");
flinkHudiOptions.put(FlinkOptions.CLEAN_POLICY.key(), HoodieCleaningPolicy.KEEP_LATEST_BY_HOURS.name());
flinkHudiOptions.put(FlinkOptions.CLEAN_RETAIN_HOURS.key(), "24");
```
My flink application works on flink-operator's FlinkDeployment (on AWS EKS).
I ran the hive-sync command once in EMR 6.10.0 (Hudi 0.12.2-amzn-0 version) for easy use of Glue MetaStore.
**To Reproduce**
Steps to reproduce the behavior:
1. run flink application with above options
2. run hive-sync once for making using hive-sync on EMR
3. run athena query when compaction is scheduled
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : 0.14.0
* Flink version : 1.16.1
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the issue with Hudi 0.14.0, Flink 1.16.1, S3, the listed MOR and asynchronous compaction options, and the one-time hive-sync setup. Run the Athena query while compaction is scheduled, inspect the referenced .hoodie/.aux compaction-requested key, and compare the result after compaction completes; done means Athena queries the MOR table successfully during compaction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100