apache / apache/doris

[Bug] Hive external table SQL cache serves stale results after out-of-band file replacement

Open
#66,133 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Description

When Hive table data files are replaced out-of-band — e.g. by an external job that only rewrites the underlying data files on HDFS/S3 without issuing an HMS DDL statement or an HMS event notification — the FE's in-memory `HMSExternalTable.updateTime` never advances.

The FE SQL cache (`NereidsSqlCacheManager` / `SqlCacheContext`, via `CacheAnalyzer`) keys cache validity on `HMSExternalTable.getUpdateTime()`. Because that value only advances from HMS table parameters (`transient_lastDdlTime`) and never from the files actually observed during a scan, a stale `updateTime` lets the SQL cache keep serving result sets computed from the old files. Queries then return stale/incorrect data until the table's metadata is next refreshed by some other trigger.

### Reproduce

1. Create a Hive external table and run a cacheable query so its result is cached (SQL cache enabled).
2. Replace the table's data files out-of-band (rewrite the files on storage) without an HMS DDL/event, keeping HMS table parameters unchanged.
3. Re-run the same query — the FE serves the cached (stale) result because `updateTime` did not advance.

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the Hive external-table scenario with SQL cache enabled, then trace cache validity through NereidsSqlCacheManager, SqlCacheContext, and CacheAnalyzer. Compare the cache key or validity timestamp with the data files observed during a scan. Done means replacing files without an HMS DDL or event no longer allows the same query to return the cached stale result.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.