Improve TLog Streaming Memory Usage
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
In release-7.1 testing an unrelated bug in simulation caused an OOM due to a large amount of TLog peek streams being opened by storage servers.
There are several memory optimizations implemented for change feed streams we could also implement for TLog streams:
- Reply immediately with an empty version message when the stream is established, instead of waiting for the first set of mutations to reply. This finishes connection establishment and lets the client+server clean up the stream more quickly
- In the merge cursor case of a single SS connecting to many TLogs, limit the memory used across the logical cursor instead of each of the individual TLog streams
- Evaluate performance vs memory tradeoffs of alternative values of SERVER_KNOBS->MAXIMUM_PEEK_BYTES
Contributor guide
Research direction
No files or tests are named. Start by locating the TLog streaming and merge cursor code, then compare the existing change feed stream memory optimizations with TLog streams. Done means validating the empty version response, shared logical-cursor memory limit, and SERVER_KNOBS->MAXIMUM_PEEK_BYTES tradeoffs without causing unacceptable performance regressions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100