tarantool / tarantool/tarantool

core: fibers engine backpressure

Open
#10,169 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature
Dominant language
Lua
Stars
3.7k
Forks
419
Avg merge
1d 23h
Merged PRs (30d)
88

Description

Recently we received feedback from a client that had some intermittent disk performance that caused significant growth in amount of requests waiting in the line to WAL write. The result was memory allocation error:

2024-06-14 13:07:18.220 [1] main fiber.c:208 !> SystemError fiber mprotect failed: Cannot allocate memory

The reason behind was actually a limitation of the memory allocation from /etc/sysctl.conf:

vm.max_map_count = 65530

To avoid further misleading we should introduce a backpressure on fiber creation. Each fiber requires two vm maps - for stack and a red zone after it. Hence, by reading the mentioned vm.max_map_count we can set a limit of 45% to the number of client_fiber_count (means 90% of available mappings) inside the `fiber_new_ex()' with appropriate report to the log in case of error. Some additional info can be provided, such as last successful xlog write, rate of the writes, etc.

Also there was a request from the client to have an ability to set the limit by hands during Tarantool instance configuration. It can be done via sysctl operation, but it is system-wide and can affect other services on the same machine.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with fiber.c:208 and the fiber_new_ex() entry point, then review how client_fiber_count and vm.max_map_count relate to fiber allocation. Define the intended backpressure limit, configuration mechanism, and log details before implementing; done means allocation is bounded and failures report useful context.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, linux
Domain
backend, operating-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.