apache / apache/hudi

[SUPPORT] Error when selecting RT table on AWS Athena (0.14.1) - with custom Payload class

Open
#11,725 1 comment 0 reactions 0 assignees View on GitHub
area:aws area:reader
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Env: AWS EMR on EKS 7.1
Hudi version: 0.14.1
Athena engine: v3
Table mode: MOR

**This happens ONLY if I use custom Payload class.**
When RT and RO table aren’t synced, selecting RT table in Athena returns the below error:
_GENERIC_INTERNAL_ERROR: Exception when constructing record reader
This query ran against the “xxx database, unless qualified by the query. Please post the error message on our forum_ 

Initially, i though that i was doing something wrong in my custom code, but then, i created the class empty, just extending the default _OverwriteWithLatestAvroPayload_, and i continue getting the error, so at this point, seems that only the fact that i'm using a Custom payload class is provoking the error (?)

I'm providing the main code that i'm using, as well as the "empty" custom payload class:
[main_code_attach.txt](https://github.com/user-attachments/files/16472848/main_code_attach.txt)
[custom_java.txt](https://github.com/user-attachments/files/16472873/custom_java.txt)

To reproduce, do one execution with BATCH 1 (commented in the main file), and do a second execution with BATCH 2.
After it, if you query the RT table in Athena, you should see the error
GENERIC_INTERNAL_ERROR: Exception when constructing record reader

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing main_code_attach.txt and custom_java.txt, then reproduce the report by running BATCH 1 followed by BATCH 2. Query the RT table in AWS Athena and compare the result with the reported “Exception when constructing record reader” error. Done means the reproduction is understood and the RT query no longer fails, or the cause is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
cloud, data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.