apache / apache/hudi

[SUPPORT] AWS Athena query fail when compaction is scheduled for MOR table

Open
#9,907 11 comments 1 reaction 0 assignees View on GitHub
area:aws area:table-service component:trino priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

I'm using hudi 0.14.0 with flink 1.16.1 to store data from kafka to s3.
but Athena(Engine 3) query to MOR table is not working because of this error.

```
Error running query: HIVE_UNKNOWN_ERROR: io.trino.plugin.hive.s3.TrinoS3FileSystem$UnrecoverableS3OperationException: com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: ***; S3 Extended Request ID: ***; Proxy: null), S3 Extended Request ID: *** (Bucket: mybucket, Key: mytable/.hoodie/.aux/20231014095517882.compaction.requested)
```

This error occurs if compaction is scheduled.
After compaction is complete, query is working.

Here's flink hudi option (Java)
```
flinkHudiOptions.put(FlinkOptions.PATH.key(), basePath);
flinkHudiOptions.put(FlinkOptions.TABLE_TYPE.key(), HoodieTableType.MERGE_ON_READ.name());
flinkHudiOptions.put(FlinkOptions.OPERATION.key(), WriteOperationType.UPSERT.name());
flinkHudiOptions.put(FlinkOptions.PRECOMBINE_FIELD.key(), "event_time");
flinkHudiOptions.put(FlinkOptions.KEYGEN_CLASS_NAME.key(), "org.apache.hudi.keygen.ComplexKeyGenerator");
flinkHudiOptions.put(FlinkOptions.COMPACTION_ASYNC_ENABLED.key(), "true");
flinkHudiOptions.put(FlinkOptions.COMPACTION_TRIGGER_STRATEGY.key(), FlinkOptions.NUM_COMMITS);
flinkHudiOptions.put(FlinkOptions.COMPACTION_DELTA_COMMITS.key(), "5");
flinkHudiOptions.put(FlinkOptions.COMPACTION_MAX_MEMORY.key(), "1024");
flinkHudiOptions.put(FlinkOptions.METADATA_ENABLED.key(), "true");
flinkHudiOptions.put(HoodieMetadataConfig.ASYNC_INDEX_ENABLE.key(), "true");
flinkHudiOptions.put(HoodieMetadataConfig.ENABLE_METADATA_INDEX_COLUMN_STATS.key(), "true");
flinkHudiOptions.put(HoodieWriteConfig.WRITE_CONCURRENCY_MODE.key(), WriteConcurrencyMode.OPTIMISTIC_CONCURRENCY_CONTROL.name());
flinkHudiOptions.put(HoodieLockConfig.LOCK_PROVIDER_CLASS_NAME.key(), "org.apache.hudi.client.transaction.lock.InProcessLockProvider");
flinkHudiOptions.put(FlinkOptions.CLEAN_ASYNC_ENABLED.key(), "true");
flinkHudiOptions.put(FlinkOptions.CLEAN_POLICY.key(), HoodieCleaningPolicy.KEEP_LATEST_BY_HOURS.name());
flinkHudiOptions.put(FlinkOptions.CLEAN_RETAIN_HOURS.key(), "24");
```

My flink application works on flink-operator's FlinkDeployment (on AWS EKS).
I ran the hive-sync command once in EMR 6.10.0 (Hudi 0.12.2-amzn-0 version) for easy use of Glue MetaStore.

**To Reproduce**

Steps to reproduce the behavior:

1. run flink application with above options
2. run hive-sync once for making using hive-sync on EMR
3. run athena query when compaction is scheduled

**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment Description**

* Hudi version : 0.14.0

* Flink version : 1.16.1

* Hive version :

* Hadoop version :

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) : no

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the issue with Hudi 0.14.0, Flink 1.16.1, S3, the listed MOR and asynchronous compaction options, and the one-time hive-sync setup. Run the Athena query while compaction is scheduled, inspect the referenced .hoodie/.aux compaction-requested key, and compare the result after compaction completes; done means Athena queries the MOR table successfully during compaction.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.