ome / ome/openmicroscopy

test_cleanse.py fails on NFS - file not released by Blitz

Open
#6,461 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
222
Forks
105
Avg merge
2h 21m
Merged PRs (30d)
1

Description

It looks like the Blitz process is keeping a filehandle on the .log file, which then is unlinked by the cleanse but is not deleted by the NFS.

cc @joshmoore @jburel @sbesson

Edit: I think the below proves that the Blitz is keeping the log file open for very long. The test is not fixable on NFS by longer waits.

pytest test/integration/clitest/test_cleanse.py::TestCleanseFullAdmin::testCleanseNonsenseName -vv -s
======================================================================================== test session starts ========================================================================================
platform linux -- Python 3.11.13, pytest-9.1.1, pluggy-1.6.0 -- /home/omero/workspace/OMERO-test-integration/.venv3/bin/python3.11
cachedir: .pytest_cache
django: version: 5.2.16, settings: omeroweb.settings (from ini)
rootdir: /home/omero/workspace/OMERO-test-integration/src/components/tools
configfile: pytest.ini
plugins: mock-3.15.1, xdist-3.8.0, django-4.12.0
collected 1 item                                                                                                                                                                                    

test/integration/clitest/test_cleanse.py::TestCleanseFullAdmin::testCleanseNonsenseName logfile_path still exists: /home/omero/workspace/OMERO-test-integration/data/ManagedRepository/f78fc6f8-b22c-4cfc-ae93-88e6d54b2d69_4826/2026-07/14/
Contents: ['.nfs000000838433c29800002cec']
FAILED

============================================================================================= FAILURES ==============================================================================================
___________________________________________________________________________ TestCleanseFullAdmin.testCleanseNonsenseName ____________________________________________________________________________

self = <test.integration.clitest.test_cleanse.TestCleanseFullAdmin object at 0x7f0fa9bc1ed0>, capsys = <_pytest.capture.CaptureFixture object at 0x7f0f9cb2a790>

    def testCleanseNonsenseName(self, capsys):
        """
        Test cleanse removes file on disk after OriginalFile
        name was changed to nonsense and its Image was deleted
        """
        # import image and retrieve the OriginalFile
        # (orig_file), its name and path
        image = self.import_fake_file()[0]
        fileset = self.get_fileset([image])
        params = omero.sys.ParametersI()
        params.addId(fileset.getId())
        q = ("select originalFile.path, originalFile.id "
             "from FilesetEntry where fileset.id = :id")
        queryService = self.root.sf.getQueryService()
        result = queryService.projection(q, params, self.group_ctx)
        path_in_mrepo = result[0][0].getValue()
        orig_file_path = self.make_path(path_in_mrepo)
        assert os.path.exists(orig_file_path)
        orig_file_id = result[0][1].getValue()
        orig_file = self.query.get("OriginalFile", orig_file_id)
        orig_file_name = orig_file.getName().getValue()
        orig_file_path_and_name = self.make_path(path_in_mrepo, orig_file_name)
        assert os.path.isfile(orig_file_path_and_name)
    
        # retrieve the logfile, its name and path
        q = ("select o from FilesetJobLink l "
             "join l.parent as fs join l.child as j "
             "join j.originalFileLinks l2 join l2.child as o "
             "where fs.id = :id and "
             "o.mimetype = 'application/omero-log-file'")
        logfile = queryService.findByQuery(q, params, self.group_ctx)
        logfile_name = logfile.getName().getValue()
        path_in_mrepo = logfile.getPath().getValue()
        logfile_path = self.make_path(path_in_mrepo)
        assert os.path.exists(logfile_path)
        logfile_path_and_name = self.make_path(path_in_mrepo, logfile_name)
        assert os.path.isfile(logfile_path_and_name)
    
        # change the names of original_file and logfile to nonsense
        name = "nonsensical"
        update_service = self.root.sf.getUpdateService()
        orig_file.setName(omero.rtypes.rstring(name))
        update_service.saveAndReturnObject(orig_file, self.group_ctx)
        logfile.setName(omero.rtypes.rstring(name))
        update_service.saveAndReturnObject(logfile, self.group_ctx)
    
        # run the cleanse command, which will not delete
        # the files on disk
        self.args += [self.data_dir]
        self.cli.invoke(self.args, strict=True)
        assert os.path.exists(orig_file_path)
        assert os.path.isfile(orig_file_path_and_name)
        assert os.path.exists(logfile_path)
        assert os.path.isfile(logfile_path_and_name)
    
        # delete the image, which will not delete
        # the files on disk because of the nonsensical name
        # of orig_file and logfile
        command = Delete2(targetObjects={"Image": [image.id.val]})
        handle = self.client.sf.submit(command)
        self.wait_on_cmd(self.client, handle)
        assert os.path.exists(orig_file_path)
        assert os.path.isfile(orig_file_path_and_name)
        assert os.path.exists(logfile_path)
        assert os.path.isfile(logfile_path_and_name)
    
        # run cleanse command again, which will now delete the
        # files on disk, the original file, the logfile
         # and their directories
        #self.cli.invoke(self.args, strict=True)
        #out, err = capsys.readouterr()
        #assert not os.path.exists(orig_file_path)
        #assert not os.path.isfile(orig_file_path_and_name)
        #assert not os.path.exists(logfile_path)
        #assert not os.path.isfile(logfile_path_and_name)
    
    
        # run cleanse command again, which will now delete the
        # files on disk, the original file, the logfile
        # and their directories
        self.cli.invoke(self.args, strict=True)
        out, err = capsys.readouterr()
    
        if os.path.exists(logfile_path):
            print(f"logfile_path still exists: {logfile_path}")
            print("Contents:", os.listdir(logfile_path))
    
        assert not os.path.exists(orig_file_path)
        assert not os.path.isfile(orig_file_path_and_name)
    
        # On NFS, an unlinked file that is still open by another process
        # may temporarily appear as a ".nfs*" file, preventing the parent
        # directory from being removed immediately. Wait briefly for any
        # such files to disappear before checking that the directory has
        # been removed.
        for _ in range(60):  # wait up to 30 seconds
            if not os.path.exists(logfile_path):
                break
    
            if not glob.glob(os.path.join(logfile_path, ".nfs*")):
                break
    
            time.sleep(0.5)
    
>       assert not os.path.exists(logfile_path)
E       AssertionError: assert not True
E        +  where True = <function exists at 0x7f103e8c7420>('/home/omero/workspace/OMERO-test-integration/data/ManagedRepository/f78fc6f8-b22c-4cfc-ae93-88e6d54b2d69_4826/2026-07/14/')
E        +    where <function exists at 0x7f103e8c7420> = <module 'posixpath' (frozen)>.exists
E        +      where <module 'posixpath' (frozen)> = os.path

test/integration/clitest/test_cleanse.py:187: AssertionError
----------------------------------------------------------------------------------------- Captured log call -----------------------------------------------------------------------------------------
INFO     omero.util.Resources:__init__.py:652 Starting
INFO     omero.util.Resources:__init__.py:669 Halted
INFO     omero.util.Resources:__init__.py:652 Starting
INFO     omero.util.Resources:__init__.py:669 Halted
========================================================================================= warnings summary ==========================================================================================
OmeroPy/test/integration/clitest/test_cleanse.py::TestCleanseFullAdmin::testCleanseNonsenseName
OmeroPy/test/integration/clitest/test_cleanse.py::TestCleanseFullAdmin::testCleanseNonsenseName
  /home/omero/workspace/OMERO-test-integration/.venv3/lib64/python3.11/site-packages/omero/plugins/sessions.py:176: DeprecationWarning: OMERO_SESSION_DIR is deprecated. Use OMERO_SESSIONDIR instead.
    warnings.warn(

OmeroPy/test/integration/clitest/test_cleanse.py::TestCleanseFullAdmin::testCleanseNonsenseName
OmeroPy/test/integration/clitest/test_cleanse.py::TestCleanseFullAdmin::testCleanseNonsenseName
OmeroPy/test/integration/clitest/test_cleanse.py::TestCleanseFullAdmin::testCleanseNonsenseName
  /home/omero/workspace/OMERO-test-integration/.venv3/lib64/python3.11/site-packages/omero/callbacks.py:270: DeprecationWarning: isSet() is deprecated, use is_set() instead
    return self.event.isSet()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
====================================================================================== short test summary info ======================================================================================
FAILED test/integration/clitest/test_cleanse.py::TestCleanseFullAdmin::testCleanseNonsenseName - AssertionError: assert not True
 +  where True = <function exists at 0x7f103e8c7420>('/home/omero/workspace/OMERO-test-integration/data/ManagedRepository/f78fc6f8-b22c-4cfc-ae93-88e6d54b2d69_4826/2026-07/14/')
 +    where <function exists at 0x7f103e8c7420> = <module 'posixpath' (frozen)>.exists
 +      where <module 'posixpath' (frozen)> = os.path
============================================================================= 1 failed, 5 warnings in 66.71s (0:01:06) ======

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with test/integration/clitest/test_cleanse.py::TestCleanseFullAdmin::testCleanseNonsenseName and reproduce it with the pytest command shown. Inspect the cleanse flow and the Blitz process involved in the test, focusing on the open .log file and the NFS .nfs* entry. Done means the test reliably removes the logfile directory on NFS without weakening the cleanup assertions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.