Stackoverflow risk when flattening a nested document
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
In the function which takes a nested document (such as a JSON document) and flattens a specific field, it's possible for a sufficiently nested document to cause a stack overflow. This is because there is no depth check at the point of recursion, so if an incoming document is structured like `{f1: {f2: f3: {...}}}` and it's deep enough, then `flattenMap` could keep recursing until there's a stack overflow.
Suggested solution: add in a depth check that will cause `flattenMap` to fault when the depth limit is reached and communicate an error message back to the user. It looks like Pinot has a standard practice of skipping incoming documents if they are malformed, so this can probably just log the error and increment the necessary meters and move to the next document.
One of the recursive calls which can lead to a stack overflow (last line in code block):
https://github.com/apache/pinot/blob/2fabbe40057d99afa497071f7083b4f16e40137a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/ComplexTypeTransformer.java#L256-L270
Contributor guide
Research direction
Start in pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/ComplexTypeTransformer.java around lines 256-270 and trace the recursive flattenMap call. Add a depth limit that reports an error instead of overflowing the stack, then follow Pinot's malformed-document handling so the document is skipped, the error is logged, and the necessary meters are incremented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100