public C API for sending callbacks to TX thread

Product: Tarantool
Since: 3.3.0
Root document: https://www.tarantool.io/en/doc/latest/reference/reference_capi/

SME: @ Gerold103

Details

There are 2 new functions in the public C API (in module.h). Their
complete description is below.

typedef void
(*tnt_tx_func_f)(void *);

/**
 * Schedule the given callback to be executed in TX thread with the provided
 * argument. In order for the messages to be actually sent to TX thread the user
 * must call tnt_tx_flush() in the same thread as the pushes.
 *
 * In TX thread the callbacks are guaranteed to start execution in the same
 * order as the push, but the order of completion is undefined if they are
 * yielding, since they get executed potentially in different fibers.
 *
 * The callbacks are handled by an internal fiber pool running in TX thread.
 * The pool has a limited size. In case the callbacks are yielding and the user
 * wants to execute more of them concurrently than the default size of the fiber
 * pool, then the size can be extended (or reduced) using
 * fiber.tx_user_pool_size() Lua API or `fiber.tx_user_pool_size` YAML config.
 *
 * If called during Tarantool shutdown, the behaviour is undefined. The external
 * threads must be terminated before that.
 *
 * If called in TX thread, the behaviour is undefined.
 *
 * The function relies on `thread_local` C++ data to have properly working
 * destructors and constructors and won't be suitable for any other runtime.
 */
API_EXPORT void
tnt_tx_push(tnt_tx_func_f func, void *arg);

/**
 * Send all the pending callbacks of this thread to TX thread. Note, that it
 * doesn't guarantee that they are already executed when this function returns.
 * They are only sent to TX thread, not called yet. It is the caller's
 * responsibility to ensure that the messages are not being sent faster than TX
 * thread is handling them. Otherwise the queue in TX thread grows faster than
 * shrinks and could lead to unpredictable latency and even OOM.
 *
 * Note, that push is very cheap while flush is relatively expensive both for
 * this thread and for TX thread. Avoid calling it on each push if possible.
 */
API_EXPORT void
tnt_tx_flush(void);

The callbacks are executed in a fiber pool which exists in TX
thread. It works exactly like another fiber pool which is used for
handling IProto requests and whose size is controlled by
box.cfg.net_msg_max.

The new pool also has a configurable size. It can be set to a
positive value using a new method in Lua fiber module:
fiber.tx_user_pool_size(new_size). The method returns the old
size. Also can be used without any arguments just to get the size:
local old_size = fiber.tx_user_pool_size().

https://www.tarantool.io/en/doc/latest/reference/reference_lua/fiber/

Another way is to configure the pool size via the YAML config.
The option is located in fiber.tx_user_pool_size. Type is
'integer'. Default is 768.

https://www.tarantool.io/en/doc/latest/reference/configuration/configuration_reference/#fiber

The users might want to extend it if they are willing to execute
more concurrent callbacks that this. Can happen, for instance, if
the API is heavily used for high load of requests doing
transactions (i.e. they are yielding a lot).
Requested by @Gerold103 in https://github.com/tarantool/tarantool/commit/d665f9586c48e5570534ea05aaf347ffc25b0a02.

Start with the C API reference linked in the issue and the public declarations in module.h. Then review the linked Lua fiber and configuration reference pages for the pool-size setting. Done means the two C functions, callback behavior, Lua method, and YAML option are documented consistently in the appropriate Tarantool reference sections.