confluentinc / confluentinc/ksql

ability to sample data from subset of partitions

Open
#409 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
315
Forks
1k
Avg merge
1d 7h
Merged PRs (30d)
36

Description

One of the cool and useful things about KSQL is the ability to "just look at some messages" to confirm structure and/or contents before building and running a continuous query which will write to an output topic. But running short interactive queries on a topic with many partitions is very slow and heavyweight in terms of setting up and then tearing down the associated streams job. Would be useful to have a way to say something like "select * from foo where bar='baz' partition 1 limit 10" and have that only query the first partition of the topic. Not sure if 'partition' is the best keyword or syntax there or not, maybe 'sample' or something else could work.

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by tracing how short interactive queries are executed over Kafka topics and how partition selection is handled, then review the existing SQL grammar before choosing syntax. Done means a supported way to sample selected partitions with a bounded result and suitable coverage for the new behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka, sql
Domain
data, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.