tarantool / tarantool/tarantool

test: verify that no yield occurs inside an ffi call

Open
#5,545 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

lua qa
Dominant language
Lua
Stars
3.7k
Forks
419
Avg merge
1d 23h
Merged PRs (30d)
88

Description

Tarantool version: 2.7.0-41-ge23b14dc2.

In fact it is duplicate of #1700, which was closed by 2711797be8c2e864db874a3f3a8434bc182d2360, but...

After the change, a code, which yields from an ffi call, may by gracefully handled, but under some conditions[^1] it may lead to panic(). I would still consider any yield from an ffi call within our codebase as a bug.

Since it may be hard to visually spot all such cases, I propose to do it mechanically. Disable JIT, save fiber()->csw at entering to an ffi call from Lua VM, compare it with the same counter at leaving the ffi call. Why to disable JIT? AFAIU, otherwise we can bypass the VM cws checks and left some ffi calls unchecked.

Rough list of activities within this task:

  • Discuss with @igormunkin how the check should be implemented.
  • Create a PoC patch for LuaJIT / Tarantool with the fiber()->cws checks.
  • Disable JIT for the whole testing (like so) and run our test suite several times.
  • Verify that the approach able to catch #4570.
  • Trackerize all found problems.
  • Decide regarding inclusion of testing of this kind into CI.
  • If we'll decide to do so:
    • Transform the PoC patch into a special build kind (add a CMake option).
    • Add an option to test-run to disable JIT for the whole testing (except, maybe tarantool that are spawn using popen from a test[^2]).
    • Add the custom build and the custom test-run option as a CI job (maybe with manual triggering).

Inspired by #4570.

[^1]: AFAIU, if the yield is not occur at a jit trace writting, but occurs at the trace execution (say, on the next iteration of a loop). So the code should yield sometimes. It is not rare case for IO. Say, when we read from a socket, we check whether new data is available and yield if there is nothing to read ATM. But we don't yield if data is here.

[^2]: Can we implement the ability to disable JIT in Tarantool based on an environment variable value? It would allow to overcome this limitation.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing test-run JIT handling in lib/server_mixins.py and the behavior described in #4570 and #1700. The proposed work includes a PoC for fiber()->csw checks, repeated test-suite runs with JIT disabled, and deciding whether custom build, test-run, and CI options are needed; completion depends on that design decision.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, lua, python
Domain
build-system, ci-cd, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.