apache / apache/amoro

[Improvement]: Support query on BaseStore

Open
#1,304 3 comments 0 reactions 0 assignees View on GitHub
good first issue not-stale type:improvement
Dominant language
Java
Stars
1.2k
Forks
395
Avg merge
4d 10h
Merged PRs (30d)
33

Description

### Search before asking

- [X] I have searched in the [issues](https://github.com/NetEase/arctic/issues?q=is%3Aissue) and found no similar issues.

### What would you like to be improved?

According to the description of Arctic [MixedFormat](https://arctic.netease.com/ch/concepts/table-formats/), a MixedFormat table consists of a Change Store and a BaseStore. Currently, both Spark engine and Trino engine support accessing the Change Store under MixedFormat, which is useful in data analysis scenarios. In order to further match the description of MixedFormat, we should also support accessing the BaseStore directly as a sub-table.

There are two scenarios for accessing the BaseStore directly:

1. when better query performance is desired and high query responsiveness is required, but lower data freshness is acceptable, the BaseStore can be queried directly;
2. when executing Timetravel, as MixedFormat does not yet support overall Timetravel, the BaseStore will be exposed as an Iceberg sub-table, allowing Timetravel operations to be performed directly on the BaseStore.

### How should we improve?

- Support to visit BaseStore via `db.table.base`
- Support query on BaseStore like `select * from db.table.base`
- Query on BaseStore should be implemented by Iceberg reader
- Should support all iceberg table query features on `db.table.base`
- Should not support any writer features on `db.table.base`
- Should support features above on Spark, Trino and Flink Batch.

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Subtasks

- #1406
- #1407

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start by tracing the existing Change Store access paths in the Spark and Trino engines, then compare how Flink Batch exposes table sub-tables. Done means `db.table.base` supports Iceberg-reader queries and timetravel without writer features across Spark, Trino, and Flink Batch.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.