apple / apple/foundationdb

Improve TLog Streaming Memory Usage

Open
#7,069 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

In release-7.1 testing an unrelated bug in simulation caused an OOM due to a large amount of TLog peek streams being opened by storage servers.
There are several memory optimizations implemented for change feed streams we could also implement for TLog streams:
- Reply immediately with an empty version message when the stream is established, instead of waiting for the first set of mutations to reply. This finishes connection establishment and lets the client+server clean up the stream more quickly
- In the merge cursor case of a single SS connecting to many TLogs, limit the memory used across the logical cursor instead of each of the individual TLog streams
- Evaluate performance vs memory tradeoffs of alternative values of SERVER_KNOBS->MAXIMUM_PEEK_BYTES

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by locating the TLog streaming and merge cursor code, then compare the existing change feed stream memory optimizations with TLog streams. Done means validating the empty version response, shared logical-cursor memory limit, and SERVER_KNOBS->MAXIMUM_PEEK_BYTES tradeoffs without causing unacceptable performance regressions.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.