infinyon / infinyon/sdf-support
feat: add support for multi-column partition keys
- Dominant language
- No language data
- Stars
- 1
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
When using [partitions](https://www.fluvio.io/sdf/concepts/services#parallel-processing-with-partitions), it may be helpful to allow using multiple columns as a partitioning key.
Currently, a key is assigned with:
```yaml
assign-key:
run: |
fn key_for_evt(evt: Evt) -> Result {
Ok(evt.hostname)
}
```
This works fine when dealing with a single key. But when we need composite keys (a key is made out of multiple columns), the only option is to concatenate the columns (e.g. `format!("{}+{}", evt.hostname, evt.timestamp)`).
This may result in slightly more complex SQL queries, E.g., if we want to get all the records that have a specific hostname:
```sql
SELECT * FROM evts WHERE (_key LIKE '{host-a+%');
```
It would be great if multi-column keys are allowed.
This could be either done as:
```yaml
assign-key:
run: |
fn keys_for_evt(evt: Evt) -> Result> {
Ok(vec![evt.hostname, evt.timestamp])
}
```
or:
```yaml
assign-key:
- name: hostname
run: |
fn evt_hostname(evt: Evt) -> Result {
Ok(evt.hostname)
}
- name: timestamp
run: |
fn evt_timestamp(evt: Evt) -> Result {
Ok(evt.timestamp)
}
```
Then queries would also be simpler:
```sql
SELECT * FROM evts WHERE hostname = 'host-a';
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing how the assign-key configuration defines partition keys and how those keys are exposed to SQL queries. Compare the proposed vector-returning and multiple-assignment forms, then verify that composite keys support direct predicates such as hostname = 'host-a' without relying on concatenated values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100