tarantool / tarantool/tarantool

macOS: make tests stable and portable

Open
#12,559 5 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

arm osx test
Dominant language
Lua
Stars
3.7k
Forks
419
Avg merge
1d 23h
Merged PRs (30d)
88

Description

While running the test suite on macOS, I encountered several tests that either consistently fail or flaky. This indicates that some tests rely on platform-specific assumptions or are not fully stable across environments.

It would be great to improve test stability and make them platform-independent where possible.

Environment:

Tarantool 3.7.0-entrypoint-218-g4e6123983f
Target: Darwin-arm64-Debug
Build options: cmake . -DCMAKE_INSTALL_PREFIX=/usr/local -DENABLE_BACKTRACE=TRUE
Compiler: AppleClang-17.0.0.17000013
C_FLAGS: -fexceptions -funwind-tables -fasynchronous-unwind-tables -fno-common -Wformat -Wformat-security -Werror=format-security -fstack-protector-strong -fPIC -fmacro-prefix-map=<...>=. -std=c11 -Wall -Wextra -Wno-gnu-alignof-expression -Wno-cast-function-type -Werror -g -ggdb -O0
CXX_FLAGS: -fexceptions -funwind-tables -fasynchronous-unwind-tables -fno-common -Wformat -Wformat-security -Werror=format-security -fstack-protector-strong -fPIC -fmacro-prefix-map=<...>=. -std=c++17 -Wall -Wextra -Wno-invalid-offsetof -Wno-gnu-alignof-expression -Wno-cast-function-type -Werror -g -ggdb -O0

System & Hardware
$ sw_vers
ProductName:		macOS
ProductVersion:		15.6.1
BuildVersion:		24G90

$ sysctl -n machdep.cpu.brand_string
Apple M4 Pro

List of problematic tests

Note: the list below is not exhaustive — it only includes the issues I was able to identify so far.

Fails
  • app/socket.test.lua https://github.com/tarantool/tarantool/pull/12561
    SO_REUSEPORT value differs from Linux: getsockopt() may return 512 instead of 1

  • metrics-luatest/tarantool_memory_metrics_test.lua https://github.com/tarantool/metrics/pull/542
    tnt_memory_virt is expected to be positive, but is 0 on non-Linux platforms

  • app-luatest/popen_test.lua https://github.com/tarantool/tarantool/pull/12562
    popen.shell(...):close() may return nil, err even though the handle is already closed

  • box-luatest/gh_11803_interface_option_test.lua https://github.com/tarantool/tarantool/pull/12563
    test assumes loopback interface name is lo, while macOS uses lo0

  • box-luatest/iproto_export_test.lua https://github.com/tarantool/tarantool/pull/12586
    server crashes during startup in application threads. Lua module resolution is broken (http.client.lib is attempted to be loaded from /, leading to cannot read /: Is a directory)

    server.log
    server | 2026-04-18 19:34:16.534 [88422] app1 init.c:618 C> builtin/http.client.lua:32: error loading module 'http.client.lib' from file '/':
    server |     cannot read /: Is a directory
    server | 2026-04-18 19:34:16.534 [88422] app1 init.c:626 C> #2 (null) (), builtin/internal.loaders.lua:245
    server | 2026-04-18 19:34:16.534 [88422] app1 init.c:626 C> #3 require (global), [C]:-1
    server | 2026-04-18 19:34:16.534 [88422] app1 init.c:626 C> #4 (null) (), builtin/http.client.lua:32
    
  • box-luatest/app_threads_test.lua https://github.com/tarantool/tarantool/pull/12586
    application threads fail to load standard Lua modules (e.g. metrics.tarantool).

    server.log
    server | 2026-04-18 19:34:15.896 [88396] app2 say.c:88 F> Error loading Lua module metrics.tarantool...: (null)
    
  • box-luatest/app_threads_module_test.lua https://github.com/tarantool/tarantool/pull/12586
    same issue as in box-luatest/app_threads_test.lua

  • config-luatest/roles_test.lua — fixed by https://github.com/tarantool/tarantool/pull/12543
    Shutdown log is matched too strictly: test expects line ending at Terminated,
    but macOS may add a suffix (e.g. Terminated: 15).

  • replication-luatest/gh_10506_spam_in_netbox_logs_test.lua https://github.com/tarantool/tarantool/pull/12569
    The test uses localhost:0 as a stand-in for a refused connection, but on macOS that address may produce a different socket error instead of Connection refused (Can't assign requested address).

Flaky / may hang
  • box-luatest/session_test.luaflaky under parallel runs https://github.com/tarantool/tarantool/pull/12565
    box.cfg({}) in an interactive child may fail with ALREADY_RUNNING: Failed to lock WAL directory .

  • app-luatest/http_client_http_version_test.lua https://github.com/tarantool/tarantool/pull/12587
    intermittent Could not connect to server before reaching the HTTP/ALPN assertion

  • app/fio.test.luaflaky under parallel runs https://github.com/tarantool/tarantool/pull/12594
    The gh-4794 section uses a fixed directory name under cwd as a custom TMPDIR. Under repeated runs, may fails with fio: File exists

  • metrics-luatest/tarantool_config_metrics_test.luaflaky under parallel runs
    Fixed port (localhost:3301) causes conflicts; metrics are read too early after config:reload().

  • replication-py/cluster.test.py
    Reuses iproto connection after failed SUBSCRIBE, which may leave it broken => Connection reset by peer.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Run the listed luatest, Lua, and Python tests on the Darwin-arm64 configuration, beginning with the unchecked cases: box-luatest/gh_11803_interface_option_test.lua, iproto_export_test.lua, app_threads_test.lua, session_test.lua, and replication-py/cluster.test.py. Review the linked pull requests and failure logs before choosing a case. Done means the affected tests pass reliably on macOS and remain stable under parallel runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
lua, macos, python
Domain
operating-systems, testing-qa
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.