apache / apache/pinot

User provided JsonPaths should be trimmed as early as possible

Open
#7,410 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

I noticed defensive calls to `String.trim` actually trimming the JSON path in a profile from a user, where these paths come from ingestion transformation configuration.

Screenshot 2021-09-08 at 11 15 08

This means that:
* the JSON path is trimmed at least once per ingested row, creating unnecessary allocation pressure
* the lookups for cached compiled `JsonPath`s always uses a new `String` object, which means the JSON path's hash code is calculated at least once per ingested row.

Cleaning JSON paths before initialising the record transformation would remedy this.

Contributor guide

Open the contributing guide

Research direction

Start by locating record transformation initialization and the ingestion path where user-provided JSON paths are trimmed and cached. Done means configured paths are cleaned before ingestion, with equivalent lookup behavior and no per-row trimming or repeated path allocation; add or update focused tests if the relevant test area is found.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data, databases
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.