[Feature] Support secondary index on `user_id` / (`user_id`, `order_date`) for detail tables used by Flink SQL
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Motivation
### Search before asking
- I searched the issues and found discussions around prefix lookup, but not an issue focused on this Flink SQL detail-table scenario.
### Motivation
In our real-time Flink SQL jobs, we use Fluss to store detail data such as `order_detail` or `order_item_detail`.
For these tables, the primary key is usually order-oriented, for example:
- `PRIMARY KEY (order_id)`
- `PRIMARY KEY (order_id, item_id)`
This primary-key design matches the write/update semantics of detail data.
However, many downstream real-time computations are user-oriented. We need to efficiently query detail rows by `user_id`, and in many cases also by `user_id + order_date`, and then perform further
computation in Flink SQL.
Typical access patterns are:
- query all detail rows for one `user_id`
- query all detail rows for one `user_id` on a given date
- use these rows in Flink SQL for real-time user-level aggregation, enrichment, or analysis
Examples of such computations:
- user-level order amount aggregation
- user-level order item count/statistics
- incremental user behavior analysis based on order details
Without a secondary index, the current options are not ideal:
- scan much more data than needed
- duplicate the same table with another primary-key layout
- move more state and complexity into Flink jobs
### Solution
We would like Fluss to support secondary indexes on non-primary-key columns, and make them usable from Flink SQL.
For our scenario, the most important requirement is:
- secondary index on `user_id`
- and/or composite secondary index on (`user_id`, `order_date`)
Example:
```sql
CREATE TABLE order_item_detail (
order_id BIGINT NOT NULL,
item_id BIGINT NOT NULL,
user_id BIGINT,
order_date DATE,
sku_id BIGINT,
amount DECIMAL(18, 2),
PRIMARY KEY (order_id, item_id) NOT ENFORCED
) WITH (
'table.secondary-indexes' = 'idx_user:user_id;idx_user_date:user_id,order_date'
);
This would allow Flink SQL jobs to directly access one user's detail data, or one user's detail data for a specific date, without changing the base table primary key.
### Expected value
This feature would make Fluss much more suitable for user-oriented real-time analytics on detail tables, especially when:
- the write path is keyed by order identifiers
- the compute path is keyed by user identifiers
- the same detail table needs multiple access paths
### Suggested phase-1 scope
A useful first phase would already be:
- support single-column and composite secondary indexes
- support equality-based lookup by secondary index
- maintain index entries on INSERT / UPDATE / DELETE
- allow Flink SQL planner/runtime to use the secondary index
It is not necessary to support all advanced index capabilities in phase 1.
Also, since Flink SQL does not currently expose CREATE INDEX, defining secondary indexes through table options would already be acceptable for us.
### Willingness to contribute
- We are happy to discuss the design further if this direction makes sense.
### Solution
_No response_
### Anything else?
_No response_
### Willingness to contribute
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no files, tests, or entry points to start from. First map the existing table-option handling, index maintenance, and Flink SQL planner/runtime integration; done means single-column and composite equality lookups are supported and indexes remain correct across INSERT, UPDATE, and DELETE.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- data-engineering, databases, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100