Use middleware mechanism to do preprocessing
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Copy from discussion at https://github.com/apache/incubator-pinot/pull/6811#issuecomment-827016380
I observe there are bunch of pre-processing logic for a request:
* authz/authn
* admission control (e.g. disallow user to create new segments if existing number of segments exceed a threshold. We don't have this logic yet, but it's useful and may be added in future)
* request validation (validate table schema, validate queries etc)
* request rewriting( e.g. this [PR](https://github.com/apache/incubator-pinot/pull/6811) rewrite request by normalizing LHS and RHS for numerical types)
* log and audit.
Now all those logic are put into BrokerRequestHandler, which is a little messy and hard to maintain. Ideally, we should use HTTP middleware to decouple those pre-processing/post-processing logic from the actual query processing logic. Most web framework provide such a HTTP middleware mechanism out of box, the HTTP library JAX-RS we use also provide similar mechanism: [Filters and Interceptors](https://dennis-xlc.gitbooks.io/restful-java-with-jax-rs-2-0-2rd-edition/content/en/part1/chapter12/filters_and_interceptors.html)
cc @Jackie-Jiang @siddharthteotia @snleee @mcvsubbu
Contributor guide
Research direction
Start by reading BrokerRequestHandler and the linked discussion and PR 6811 to map the existing preprocessing responsibilities. Compare the JAX-RS Filters and Interceptors mechanism described in the issue, then define the middleware scope and acceptance criteria for separating preprocessing and post-processing from query handling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100