apache / apache/druid

Be able to define batchSize for SQL requests

Open
#14,215 1 comment 0 reactions 0 assignees View on GitHub
Design Review Proposal
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Motivation

Currently, it is possible to do scan queries in both native and SQL queries.
Nevertheless, it is possible to define batchSize only in native queries. In SQL queries the value is the default value, with no possibility to change its value.

### Proposed changes

- SQL language:

In SQL context parameters, allow to add a non mandatory `batchSize` parameter.
This parameter is a positive Integer, similarly to the same parameter in native scan queries.
Default value is 20480.
During SQL query translation to native scan query, the `batchSize` parameter found in SQL context will be used in the generated native scan query for its batchSize parameter value.

### Rationale

This feature allows users that generate complex druid queries On-the-fly from various parameters in their applications to avoid to have to handle both SQL and native query generation in their code, just to be able to handle scan queries which needs controlled batch size.
Dealing with both native and SQL query generation in code can leads to high complexity.

### Operational impact

This should have no effect on existing clusters.

Contributor guide

Open the contributing guide

Research direction

Start by tracing SQL context parameter handling and the SQL-to-native scan query translation described in the issue. Verify how native scan queries validate and default batchSize, then identify the existing SQL query tests. Done means SQL accepts an optional positive batchSize, defaults to 20480, and passes the value into the generated native scan query without changing existing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
database
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.