linkedin / linkedin/Hoptimator
Expose "hints" to the planner
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 128
- Forks
- 16
- Avg merge
- 5h 5m
- Merged PRs (30d)
- 4
Description
#37 adds "subscription hints", which are passed to the control plane via templates. This is useful for things like kafka.numPartitions which are specific to the control plane.
For more advanced use-cases, we need the planner (i.e. PipelineRel etc) to have access to "hints" as well. This would enable users to influence the output (Flink) SQL generated for a pipeline. For example, hints could be used to dynamically enable/disable specific planner rules.
Calcite has a concept of "contexts" which might be useful here. See: https://calcite.apache.org/javadocAggregate/org/apache/calcite/plan/Context.html
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the existing subscription-hint flow from #37 and inspect the planner entry points such as PipelineRel. Read Calcite's Context documentation to evaluate how hints could reach the planner. Done means planner components can access subscription hints and use them to influence generated Flink SQL, including planner-rule selection.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka, sql
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100