apache / apache/kyuubi

faced "No such file or directory " when query hudi

Open
#5,119 7 comments 0 reactions 0 assignees View on GitHub
kind:bug priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.

### Describe the bug

spark 3.3.2
hudi:0.13.1
kyuubi:1.7.0

Whey query the hudi tables used to get below error messages even refresh tables:

No such file or directory: s3a://xxxx/3cb781bd-4efc-4277-9b35-6d436f39ddc1-0_24-373672-1708766_20230727133636.parquet

### Affects Version(s)

1.7.0

### Kyuubi Server Log Output

_No response_

### Kyuubi Engine Log Output

_No response_

### Kyuubi Server Configurations

```yaml
kyuubi.frontend.rest.bind.host 0.0.0.0
kyuubi.frontend.rest.bind.port 10099
kyuubi.frontend.protocols=THRIFT_BINARY,REST

kyuubi.credentials.hadoopfs.enabled=false
kyuubi.ha.enabled=true
kyuubi.ha.zookeeper.quorum=xxxx
kyuubi.ha.zookeeper.namespace=kyuubi_cluster001
spark.master=yarn
spark.submit.deployMode=cluster

#hudi
spark.serializer=org.apache.spark.serializer.KryoSerializer
spark.sql.extensions==org.apache.kyuubi.sql.KyuubiSparkSQLExtension,org.apache.spark.sql.hudi.HoodieSparkSessionExtension
spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog

spark.hive.metastore.uris=ehrift://xxxx:9083
spark.yarn.jars=xxx/spark3jars
spark.yarn.archive=xxxx/spark3_jars.zip
spark.sql.legacy.parquet.datetimeRebaseModeInRead=LEGACY
spark.sql.legacy.timeParserPolicy=LEGACY
spark.sql.session.timeZone=UTC
kyuubi.session.engine.idle.timeout=PT300M
kyuubi.session.engine.initialize.timeout=PT20M

#kyuubi
spark.sql.optimizer.insertRepartitionBeforewrite.enable=true

# AQE
spark.sql.adaptive.enabled=true
spark.sql.adaptive.advisoryPartitionSizeInBytes=64M
spark.sql.adaptive.coalescePartitions.minPartitionNum=1

# spark
spark.executor.memory=6G
spark.driver.memory=8G
spark.executor.cores=1
spark.sql.autoBroadcastJoinThreshold=20971520
spark.sql.adaptive.autoBroadcastJoinThreshold=20971520
spark.sql.broadcastTimeout=1200
spark.sql.legacy.parquet.datetimeRebaseModeInRead=LEGACY
spark.sql.legacy.timeParserPolicy=LEGACY
spark.sql.parquet.int96RebaseModeInWrite=LEGACY
spark.sql.hive.convertMetastoreParquet=true
spark.memory.fraction=0.8
spark.memory.storageFraction=0.3

# Overhead
spark.executor.memoryOverhead=2G

spark.driver.memoryOverhead=1G
spark.driver.maxResultSize=1G

#spark.memory.offHeap.enabled=true
#spark.memory.offHeap.size=2G

spark.dynamicAllocation.enabled=true
spark.shuffle.service.enabled=true
spark.dynamicAllocation.minExecutors=6
spark.dynamicAllocation.maxExecutors=60
spark.dynamicAllocation.executorAllocationRatio=0.5
spark.dynamicAllocation.sustainedSchedulerBacklogTimeout=5s
spark.dynamicAllocation.cachedExecutorIdleTimeout=30min
spark.dynamicAllocation.executorIdleTimeout=60s

# gc
spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1ReservePercent=15 -XX:+DisableExplicitGC -Duser.timezone=UTC
spark.executor.extraJavaOptions=-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1ReservePercent=15 -XX:+DisableExplicitGC -Duser.timezone=UTC
#
```

### Kyuubi Engine Configurations

_No response_

### Additional context

_No response_

### Are you willing to submit PR?

- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.

Contributor guide

Open the contributing guide

Research direction

Reproduce the Hudi query using Spark 3.3.2, Hudi 0.13.1, and Kyuubi 1.7.0 with the listed configurations. Start by collecting the missing Kyuubi server and engine logs and checking why the referenced s3a parquet file is unavailable; done means the cause is identified and a verified fix or configuration change is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.