apple / apple/foundationdb

Provide controls to control C Client's long term memory usage

Open
#1,289 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

Please refer to the discussion in the forum: [link](https://forums.foundationdb.org/t/fdb-c-client-library-memory-allocations/1199): It appears the there is no cap on the maximum amount of memory that can be allocated by fast-allocator in the C client library. This makes it hard to reason about how much memory to allocate to the FDB client application (accounting for fdb client lib + core application memory requirements).

We could try to give the process an upper bound on memory usage (using OS constructs, that kills the process if it goes above this cap), but that may not be very effective, given that the C pool may eat up memory from the process's core functionality. This is also quite fragile due to inability to deal with temporary spikes in memory usage.

I was thinking, as a crude suggestion, to configure a "long-term" memory pool size for the client library. Any memory allocated over this size is freed up immediately when the transaction is completed. This would help the users control maximum memory that can be held by fdb client lib for a long-term, while being able to accommodate spikes of request/response that require more memory momentarily.

If we also log any such deallocations in the client library, the users can analyze those and make a decision whether to increase the long-term max pool size or incur the runtime cost of allocations/deallocations in situations when pool does not have enough memory.

Contributor guide

Open the contributing guide

Research direction

Start with the linked forum discussion and the C client library's fast-allocator, which are the only implementation pointers provided. Define how a configurable long-term pool limit and logging should behave, including temporary allocation spikes, then validate the design with focused client-library tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, cpp
Domain
databases, distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.