apache / apache/fluss

[Feature] Support secondary index on `user_id` / (`user_id`, `order_date`) for detail tables used by Flink SQL

Open
#3,064 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Motivation

### Search before asking

- I searched the issues and found discussions around prefix lookup, but not an issue focused on this Flink SQL detail-table scenario.

### Motivation

In our real-time Flink SQL jobs, we use Fluss to store detail data such as `order_detail` or `order_item_detail`.

For these tables, the primary key is usually order-oriented, for example:

- `PRIMARY KEY (order_id)`
- `PRIMARY KEY (order_id, item_id)`

This primary-key design matches the write/update semantics of detail data.

However, many downstream real-time computations are user-oriented. We need to efficiently query detail rows by `user_id`, and in many cases also by `user_id + order_date`, and then perform further
computation in Flink SQL.

Typical access patterns are:

- query all detail rows for one `user_id`
- query all detail rows for one `user_id` on a given date
- use these rows in Flink SQL for real-time user-level aggregation, enrichment, or analysis

Examples of such computations:

- user-level order amount aggregation
- user-level order item count/statistics
- incremental user behavior analysis based on order details

Without a secondary index, the current options are not ideal:

- scan much more data than needed
- duplicate the same table with another primary-key layout
- move more state and complexity into Flink jobs

### Solution

We would like Fluss to support secondary indexes on non-primary-key columns, and make them usable from Flink SQL.

For our scenario, the most important requirement is:

- secondary index on `user_id`
- and/or composite secondary index on (`user_id`, `order_date`)

Example:

```sql
CREATE TABLE order_item_detail (
order_id BIGINT NOT NULL,
item_id BIGINT NOT NULL,
user_id BIGINT,
order_date DATE,
sku_id BIGINT,
amount DECIMAL(18, 2),
PRIMARY KEY (order_id, item_id) NOT ENFORCED
) WITH (
'table.secondary-indexes' = 'idx_user:user_id;idx_user_date:user_id,order_date'
);

This would allow Flink SQL jobs to directly access one user's detail data, or one user's detail data for a specific date, without changing the base table primary key.

### Expected value

This feature would make Fluss much more suitable for user-oriented real-time analytics on detail tables, especially when:

- the write path is keyed by order identifiers
- the compute path is keyed by user identifiers
- the same detail table needs multiple access paths

### Suggested phase-1 scope

A useful first phase would already be:

- support single-column and composite secondary indexes
- support equality-based lookup by secondary index
- maintain index entries on INSERT / UPDATE / DELETE
- allow Flink SQL planner/runtime to use the secondary index

It is not necessary to support all advanced index capabilities in phase 1.

Also, since Flink SQL does not currently expose CREATE INDEX, defining secondary indexes through table options would already be acceptable for us.

### Willingness to contribute

- We are happy to discuss the design further if this direction makes sense.

### Solution

_No response_

### Anything else?

_No response_

### Willingness to contribute

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no files, tests, or entry points to start from. First map the existing table-option handling, index maintenance, and Flink SQL planner/runtime integration; done means single-column and composite equality lookups are supported and indexes remain correct across INSERT, UPDATE, and DELETE.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
data-engineering, databases, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.