PQ could benefit by avoiding deserialization cost
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 88
Description
While work has recently been done to pull PQ deserialization out of the lock by deferring the reification of events in a batch until after the lock has been released, we could further benefit by avoiding that deserialization cost entirely in the absence of memory pressure.
TODO: flesh out the requirements, using the [proof-of-concept](https://github.com/elastic/logstash/pull/17820) as a starting point.
- held events should be _soft_ references to allow GC to collect them if doing so can prevent an OOM
- the soft-references should be removed after a hard-reference has been obtained and before said hard-reference becomes part of a batch
- when a page is deactivated (and its memory-mapped file released), its cache of events should also be deleted
- the local event cache should be feature-gated for safety (even if we are confident launching it as opt-out)
Contributor guide
Research direction
Start by reviewing the linked proof-of-concept pull request and use it to flesh out the requirements. Done should cover soft references that can be cleared under memory pressure, safe reference removal before batching, cache deletion when a page is deactivated, and a feature gate for the local event cache.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100