tarantool / tarantool/tt

tests: running cluster flacky test

Open
#974 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

1sp flaky test teamE
Dominant language
Go
Stars
113
Forks
18
Avg merge
4d 11h
Merged PRs (30d)
23

Description

We have a flacky test in test/integration/running/test_running.py::test_running_env_variables:

See the error log
tt_cmd = PosixPath('/private/tmp/pytest-of-tntmac04.tarantool.i/pytest-19/tt_build0/tt')
tmpdir_with_cfg = '/private/tmp/pytest-of-tntmac04.tarantool.i/pytest-19/test_running_env_variables0'

    def test_running_env_variables(tt_cmd, tmpdir_with_cfg):
        tmpdir = tmpdir_with_cfg
        # Copy the test application to the "run" directory.
        test_app_path = os.path.join(os.path.dirname(__file__), "test_env_app", "test_env_app.lua")
        shutil.copy(test_app_path, tmpdir)
    
        # Set environmental variable which changes log format to json.
        my_env = os.environ.copy()
        my_env["TT_LOG_FORMAT"] = "json"
    
        try:
            # Start an instance with custom env.
            start_cmd = [tt_cmd, "start", "test_env_app"]
            instance_process = subprocess.Popen(
                start_cmd,
                cwd=tmpdir,
                stderr=subprocess.STDOUT,
                stdout=subprocess.PIPE,
                text=True,
                env=my_env
            )
            start_output = instance_process.stdout.readline()
            assert re.search(r"Starting an instance", start_output)
    
            # Check status.
            file = wait_file(os.path.join(tmpdir, "test_env_app", run_path, "test_env_app"),
                             pid_file, [])
            assert file != ""
            status_cmd = [tt_cmd, "status", "test_env_app"]
            status_rc, status_out = run_command_and_get_output(status_cmd, cwd=tmpdir)
            assert status_rc == 0
            status_out = extract_status(status_out)
            assert status_out["test_env_app"]["STATUS"] == "RUNNING"
    
            # Check that log format is in json.
            logPath = os.path.join(tmpdir, "test_env_app", "var", "log", "test_env_app", log_file)
            wait_string_in_file(logPath, "{")
        finally:
            # Stop the Instance.
            stop_cmd = [tt_cmd, "stop", "-y", "test_env_app"]
            stop_rc, stop_out = run_command_and_get_output(stop_cmd, cwd=tmpdir)
            assert stop_rc == 0
>           assert re.search(r"The Instance test_env_app \(PID = \d+\) has been terminated.", stop_out)
E           AssertionError: assert None
E            +  where None = <function search at 0x104a6d5a0>('The Instance test_env_app \\(PID = \\d+\\) has been terminated.', '')
E            +    where <function search at 0x104a6d5a0> = re.search

test/integration/running/test_running.py:618: AssertionError
----------------------------- Captured stdout call -----------------------------
 INSTANCE      STATUS   PID    MODE  CONFIG  BOX  UPSTREAM 
 test_env_app  RUNNING  77940        --      --   --       

The status of some instances requires attention.
Please rerun the command with the --details flag to see more information


------------------------------ Captured log call -------------------------------
WARNING  retry.api:api.py:40 , retrying in 0.5 seconds...
=========================== short test summary info ============================
FAILED test/integration/running/test_running.py::test_running_env_variables - AssertionError: assert None
 +  where None = <function search at 0x104a6d5a0>('The Instance test_env_app \\(PID = \\d+\\) has been terminated.', '')
 +    where <function search at 0x104a6d5a0> = re.search

This arises because the instance does not have time to stop in time. We need to wait for the stopping to get an expected log message.

The test with the similar error of waiting fails too here: https://github.com/tarantool/tt/issues/972.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with test/integration/running/test_running.py::test_running_env_variables and compare the similar waiting failure in issue 972. Run the named test and inspect the stop cleanup path; done means the instance has time to stop and the expected termination message is available for the assertion.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, testing
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.