[bug] Repeating hudi_table_changes query on the same large table gets stuck
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 112
Description
**Describe the problem you faced**
**To Reproduce**
Steps to reproduce the behavior:
1.
spark-sql --packages org.apache.hudi:hudi-spark3.2-bundle_2.12:0.14.0 --master yarn --driver-memory 8g --num-executors 10 --conf spark.dynamicAllocation.maxExecutors=20 --executor-memory 4G --executor-cores 2 --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog --conf spark.kryo.registrator=org.apache.spark.HoodieSparkKryoRegistrar --conf spark.sql.autoBroadcastJoinThreshold=2G --conf spark.memory.storageFraction=0.5 --conf spark.sql.broadcastTimeout=60000 --conf spark.yarn.priority=5 --conf spark.sql.broadcastTimeout=600000 --conf spark.network.timeout=600000s --conf spark.eventLog.enable=false --conf spark.driver.maxResultSize=4g --conf spark.driver.extraJavaOptions=-XX:-UseGCOverheadLimit --conf spark.executor.extraJavaOptions=-XX:-UseGCOverheadLimit --name zyc_test --conf spark.dynamicAllocation.enabled=false
2.SELECT count(1) FROM hudi_table_changes('bi_ods_real.ods_log_smart_datapoint_report_batch_rt', 'latest_state', '20231114033500000', '20231114040500000');
Results can be returned normally
3. SELECT count(1) FROM hudi_table_changes('bi_ods_real.ods_log_smart_datapoint_report_batch_rt', 'latest_state', '20231114033500000', '20231114040500000');
It's stuck and won't exit. It's hard to exit by Ctrl+C.

**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version :0.14.0
* Spark version :3.2.1
* Hive version :3.1.3
* Hadoop version :3.2.2
* Storage (HDFS/S3/GCS..) :s3
* Running on Docker? (yes/no) :no
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the issue with the supplied spark-sql configuration and the two identical hudi_table_changes queries against the S3-backed table. Trace the table-changes query path to determine why the second count remains stuck; done means repeated execution returns normally instead of hanging.
Written by the indexing model from the issue text.
Assessment
- Domain
- cloud, data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100