elastic / elastic/logstash

PQ could benefit by avoiding deserialization cost

Open
#17,821 0 comments 0 reactions 0 assignees View on GitHub
enhancement status:needs-triage
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

While work has recently been done to pull PQ deserialization out of the lock by deferring the reification of events in a batch until after the lock has been released, we could further benefit by avoiding that deserialization cost entirely in the absence of memory pressure.

TODO: flesh out the requirements, using the [proof-of-concept](https://github.com/elastic/logstash/pull/17820) as a starting point.

- held events should be _soft_ references to allow GC to collect them if doing so can prevent an OOM
- the soft-references should be removed after a hard-reference has been obtained and before said hard-reference becomes part of a batch
- when a page is deactivated (and its memory-mapped file released), its cache of events should also be deleted
- the local event cache should be feature-gated for safety (even if we are confident launching it as opt-out)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the linked proof-of-concept pull request and use it to flesh out the requirements. Done should cover soft references that can be cleared under memory pressure, safe reference removal before batching, cache deletion when a page is deactivated, and a feature gate for the local event cache.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.