Remove Bindable query path
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
Remove the Bindable query path, meaning all SQL queries can be executed as native queries (or a sequence of concatenated native queries, in the case of UNION ALL). Currently, this path is used for the `sys` and `INFORMATION_SCHEMA` schemas. It is also used when the planner swaps in static `Values` for a table scan, which happens when it realizes that a `WHERE` filter can never possibly match anything.
This would allow supporting special Druid functions, like `TIME_PARSE`, on these tables. IMO this is the main benefit. There are also side benefits: like simplifying the SQL planning process, and joining between system tables and regular tables.
The approach that comes to mind first is to have the SQL layer generate lazy inline datasources for these special tables using `InlineDataSource.fromIterable`. We need to make sure we don't blow through memory or other resources limits while doing this. (The current Bindable stuff has no resource-limiting facility, so this would be an improvement.)
I believe the main prerequisite here is supporting `ORDER BY` on non-aggregating queries, meaning adding sort-by-non-time to the Scan query. It's possible that we could start this off by only supporting it for cases where there is just a single segment involved in the query. I believe this would cover the `sys` and `INFORMATION_SCHEMA` cases.
Contributor guide
Research direction
Start by tracing the SQL layer's Bindable query path and the Scan query prerequisite described in the issue, including system and INFORMATION_SCHEMA handling and static Values substitution. Done means all SQL queries use native queries or concatenated native queries for UNION ALL, support the required non-aggregating ORDER BY behavior, and enforce resource limits for inline datasources.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- backend-api-design, databases
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100