elastic / elastic/logstash

Decouple write lock and read lock of PQ

Open
#16,158 0 comments 1 reaction 0 assignees View on GitHub
performance improvements persistent queues Team:Logstash
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

Currently, every [write](https://github.com/elastic/logstash/blob/8.13/logstash-core/src/main/java/org/logstash/ackedqueue/Queue.java#L420) and read [action](https://github.com/elastic/logstash/blob/8.13/logstash-core/src/main/java/org/logstash/ackedqueue/Queue.java#L617) shares the same lock. Write action always write to the head page, while the read action could read from the tail or head page depending on the workers speed.

In case of reading the tail page, read action no need to wait the write side to release the lock to do read. Head page and tail page do not share anything and can work parallel.

Contributor guide

Open the contributing guide

Research direction

Start in logstash-core/src/main/java/org/logstash/ackedqueue/Queue.java at the write action around line 420 and read action around line 617, and trace which page state each operation accesses. The work is done when reads from the tail page no longer wait for writes to the head page, while shared state remains safely synchronized.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.