apache / apache/pinot

Use middleware mechanism to do preprocessing

Open
#6,882 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Copy from discussion at https://github.com/apache/incubator-pinot/pull/6811#issuecomment-827016380

I observe there are bunch of pre-processing logic for a request:

* authz/authn
* admission control (e.g. disallow user to create new segments if existing number of segments exceed a threshold. We don't have this logic yet, but it's useful and may be added in future)
* request validation (validate table schema, validate queries etc)
* request rewriting( e.g. this [PR](https://github.com/apache/incubator-pinot/pull/6811) rewrite request by normalizing LHS and RHS for numerical types)
* log and audit.

Now all those logic are put into BrokerRequestHandler, which is a little messy and hard to maintain. Ideally, we should use HTTP middleware to decouple those pre-processing/post-processing logic from the actual query processing logic. Most web framework provide such a HTTP middleware mechanism out of box, the HTTP library JAX-RS we use also provide similar mechanism: [Filters and Interceptors](https://dennis-xlc.gitbooks.io/restful-java-with-jax-rs-2-0-2rd-edition/content/en/part1/chapter12/filters_and_interceptors.html)

cc @Jackie-Jiang @siddharthteotia @snleee @mcvsubbu

Contributor guide

Open the contributing guide

Research direction

Start by reading BrokerRequestHandler and the linked discussion and PR 6811 to map the existing preprocessing responsibilities. Compare the JAX-RS Filters and Interceptors mechanism described in the issue, then define the middleware scope and acceptance criteria for separating preprocessing and post-processing from query handling.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.