Spark SQL re-lists Hudi table after every SQL operations
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Currently, after most DML operations in Spark SQL, Hudi invokes `Catalog.refreshTable`
Prior to Spark 3.2, this was essentially doing the following:
# Invalidating relation cache (forcing next time for relation to be re-resolved, creating new FileIndex, listing files, etc)
# Trigger cascading invalidation (re-caching) of the cached data (in CacheManager)
As of Spark 3.2 it now additionally does `LogicalRelation.refresh` for ALL tables (previously this was only done for Temporary Views), therefore entailing whole table to be re-listed again by triggering `FileIndex.refresh` which might be costly operation.
We should revert back to preceding behavior from Spark 3.1
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5697
- Type: Bug
- Epic: https://issues.apache.org/jira/browse/HUDI-3249
- Fix version(s):
- 1.1.0
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.