tarantool / tarantool/tarantool
core: fibers engine backpressure
Nobody has claimed this yet.
- Dominant language
- Lua
- Stars
- 3.7k
- Forks
- 419
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 88
Description
Recently we received feedback from a client that had some intermittent disk performance that caused significant growth in amount of requests waiting in the line to WAL write. The result was memory allocation error:
2024-06-14 13:07:18.220 [1] main fiber.c:208 !> SystemError fiber mprotect failed: Cannot allocate memory
The reason behind was actually a limitation of the memory allocation from /etc/sysctl.conf:
vm.max_map_count = 65530
To avoid further misleading we should introduce a backpressure on fiber creation. Each fiber requires two vm maps - for stack and a red zone after it. Hence, by reading the mentioned vm.max_map_count we can set a limit of 45% to the number of client_fiber_count (means 90% of available mappings) inside the `fiber_new_ex()' with appropriate report to the log in case of error. Some additional info can be provided, such as last successful xlog write, rate of the writes, etc.
Also there was a request from the client to have an ability to set the limit by hands during Tarantool instance configuration. It can be done via sysctl operation, but it is system-wide and can affect other services on the same machine.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with fiber.c:208 and the fiber_new_ex() entry point, then review how client_fiber_count and vm.max_map_count relate to fiber allocation. Define the intended backpressure limit, configuration mechanism, and log details before implementing; done means allocation is bounded and failures report useful context.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, linux
- Domain
- backend, operating-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100