[Improvement]: Support query on BaseStore
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 395
- Avg merge
- 4d 10h
- Merged PRs (30d)
- 33
Description
### Search before asking
- [X] I have searched in the [issues](https://github.com/NetEase/arctic/issues?q=is%3Aissue) and found no similar issues.
### What would you like to be improved?
According to the description of Arctic [MixedFormat](https://arctic.netease.com/ch/concepts/table-formats/), a MixedFormat table consists of a Change Store and a BaseStore. Currently, both Spark engine and Trino engine support accessing the Change Store under MixedFormat, which is useful in data analysis scenarios. In order to further match the description of MixedFormat, we should also support accessing the BaseStore directly as a sub-table.
There are two scenarios for accessing the BaseStore directly:
1. when better query performance is desired and high query responsiveness is required, but lower data freshness is acceptable, the BaseStore can be queried directly;
2. when executing Timetravel, as MixedFormat does not yet support overall Timetravel, the BaseStore will be exposed as an Iceberg sub-table, allowing Timetravel operations to be performed directly on the BaseStore.
### How should we improve?
- Support to visit BaseStore via `db.table.base`
- Support query on BaseStore like `select * from db.table.base`
- Query on BaseStore should be implemented by Iceberg reader
- Should support all iceberg table query features on `db.table.base`
- Should not support any writer features on `db.table.base`
- Should support features above on Spark, Trino and Flink Batch.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Subtasks
- #1406
- #1407
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start by tracing the existing Change Store access paths in the Spark and Trino engines, then compare how Flink Batch exposes table sub-tables. Done means `db.table.base` supports Iceberg-reader queries and timetravel without writer features across Spark, Trino, and Flink Batch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100