Provide controls to control C Client's long term memory usage
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
Please refer to the discussion in the forum: [link](https://forums.foundationdb.org/t/fdb-c-client-library-memory-allocations/1199): It appears the there is no cap on the maximum amount of memory that can be allocated by fast-allocator in the C client library. This makes it hard to reason about how much memory to allocate to the FDB client application (accounting for fdb client lib + core application memory requirements).
We could try to give the process an upper bound on memory usage (using OS constructs, that kills the process if it goes above this cap), but that may not be very effective, given that the C pool may eat up memory from the process's core functionality. This is also quite fragile due to inability to deal with temporary spikes in memory usage.
I was thinking, as a crude suggestion, to configure a "long-term" memory pool size for the client library. Any memory allocated over this size is freed up immediately when the transaction is completed. This would help the users control maximum memory that can be held by fdb client lib for a long-term, while being able to accommodate spikes of request/response that require more memory momentarily.
If we also log any such deallocations in the client library, the users can analyze those and make a decision whether to increase the long-term max pool size or incur the runtime cost of allocations/deallocations in situations when pool does not have enough memory.
Contributor guide
Research direction
Start with the linked forum discussion and the C client library's fast-allocator, which are the only implementation pointers provided. Define how a configurable long-term pool limit and logging should behave, including temporary allocation spikes, then validate the design with focused client-library tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, cpp
- Domain
- databases, distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100