apache / apache/kyuubi

[Improvement] Table Relation Cache Feature Should be Configureable in Kyuubi Server

Open
#2,857 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues?q=is%3Aissue) and found no similar issues.

### What would you like to be improved?

My scenario is to use kyuubi as a replacement for hiveserver2. When kyuubi and hive are used at the same time, the same table may be modified by two different execution engines at the same time. Due to the metadata caching feature of SparkSQL, SparkSQLEngine cannot perceive the changes of the table in time.

For example, Kyuubi user use SparkSQL Engine query a table at fisrt, then hive user insert some record to it, back to the SparkSQL Engine, the kyuubi user query the table again, they will found nothing changes.
Another example is, when Kyuubi user query a table at first, then the hive user truncate it, when the kyuubi user query this table again, SparkSQL throws exceptions as follow:
```shell
It is possible the underlying files have been updated. You can explicitly invalidate
the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by
recreating the Dataset/DataFrame involved.
```

### How should we improve?

Although there is many ways to solove this problem, such as refresh table before query it, set `spark.sql.filesourceTableRelationCacheSize` to zero when open a kyuubi session, these methods are not user friendly.
We should provide a configuration to tooggle the `TableRelationCache` feature. Maybe in many scenarios, Kyuubi admin will choose to turn off the SparkSQL table relation cache feature to reduce user complaints.

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by tracing how Kyuubi configures the SparkSQL engine and session-level table relation caching, then identify the existing configuration tests. Done means an administrator can toggle the cache through Kyuubi configuration and queries reflect external table changes as described.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark, sql
Domain
backend, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.