[SUPPORT] Error when selecting RT table on AWS Athena (0.14.1) - with custom Payload class
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Env: AWS EMR on EKS 7.1
Hudi version: 0.14.1
Athena engine: v3
Table mode: MOR
**This happens ONLY if I use custom Payload class.**
When RT and RO table aren’t synced, selecting RT table in Athena returns the below error:
_GENERIC_INTERNAL_ERROR: Exception when constructing record reader
This query ran against the “xxx database, unless qualified by the query. Please post the error message on our forum_
Initially, i though that i was doing something wrong in my custom code, but then, i created the class empty, just extending the default _OverwriteWithLatestAvroPayload_, and i continue getting the error, so at this point, seems that only the fact that i'm using a Custom payload class is provoking the error (?)
I'm providing the main code that i'm using, as well as the "empty" custom payload class:
[main_code_attach.txt](https://github.com/user-attachments/files/16472848/main_code_attach.txt)
[custom_java.txt](https://github.com/user-attachments/files/16472873/custom_java.txt)
To reproduce, do one execution with BATCH 1 (commented in the main file), and do a second execution with BATCH 2.
After it, if you query the RT table in Athena, you should see the error
GENERIC_INTERNAL_ERROR: Exception when constructing record reader
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing main_code_attach.txt and custom_java.txt, then reproduce the report by running BATCH 1 followed by BATCH 2. Query the RT table in AWS Athena and compare the result with the reported “Exception when constructing record reader” error. Done means the reproduction is understood and the RT query no longer fails, or the cause is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- cloud, data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100