infinyon / infinyon/sdf-support

feat: add support for multi-column partition keys

Open
#3 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Description

When using [partitions](https://www.fluvio.io/sdf/concepts/services#parallel-processing-with-partitions), it may be helpful to allow using multiple columns as a partitioning key.

Currently, a key is assigned with:

```yaml
assign-key:
run: |
fn key_for_evt(evt: Evt) -> Result {
Ok(evt.hostname)
}
```

This works fine when dealing with a single key. But when we need composite keys (a key is made out of multiple columns), the only option is to concatenate the columns (e.g. `format!("{}+{}", evt.hostname, evt.timestamp)`).

This may result in slightly more complex SQL queries, E.g., if we want to get all the records that have a specific hostname:

```sql
SELECT * FROM evts WHERE (_key LIKE '{host-a+%');
```

It would be great if multi-column keys are allowed.

This could be either done as:

```yaml
assign-key:
run: |
fn keys_for_evt(evt: Evt) -> Result> {
Ok(vec![evt.hostname, evt.timestamp])
}
```

or:

```yaml
assign-key:
- name: hostname
run: |
fn evt_hostname(evt: Evt) -> Result {
Ok(evt.hostname)
}
- name: timestamp
run: |
fn evt_timestamp(evt: Evt) -> Result {
Ok(evt.timestamp)
}
```

Then queries would also be simpler:

```sql
SELECT * FROM evts WHERE hostname = 'host-a';
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing how the assign-key configuration defines partition keys and how those keys are exposed to SQL queries. Compare the proposed vector-returning and multiple-assignment forms, then verify that composite keys support direct predicates such as hostname = 'host-a' without relying on concatenated values.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.