Merge CI testintegration failures - NFS

Issues

After the move to NFS this month, we are experiencing new test failures, cause typically by 2 problems, see e.g. https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-test-integration/16/testReport/

• pyramids are not being built fast enough (ca 80% of failing tests)
• mismatches of expected numbers of objects on OmeroWeb tests (ca 20% of failing tests)

Solutions - pyramids: polling

• Introduced newly: pyramids can be polled for in the test for "readiness" by polling in a loop for
  • for the presence of the file on disk, see e.g. https://github.com/ome/openmicroscopy/pull/6467/changes (the wait_for_pyramid_file method)
  • for the "renderableness" of the pyramid, see e.g. https://github.com/ome/openmicroscopy/pull/6469/changes/2607cba1f4c501e8174bcc01b0879515b393818b)

Present polling/waiting methods - pyramids

• the present method wait_for_pyramid in omero-py is already used in many of the problematic tests and proven insufficient

Solutions - OmeroWeb tests

• only suggestions were made, no direct coding - I believe @will-moore is going to get into the problematics
  • do not assume a clean state of the DB when the test starts
  • watch out for race conditions, e.g. between the chgrp server-side process and the method in json API which hastily fetches the "new" state inside the test

Caveats of polling

• even the polling loop has to have a default timeout - usually I was setting 2 mins:
  • the fixed tests are running longer (but actually this is much less of an issue when watching average times of the whole suite)
  • The timeout comes into play, because even 2 mins are (often) not enough to build the pyramid, see eg. https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-test-integration/18/testReport/OmeroPy.test.integration.test_thumbs/TestThumbs/testThumbnailVersion_one_/

Alternative solutions to polling - pyramids

• marking the tests which are problematic as "local only" (there is a precedens). I think this is a very attractive solution, as the pyramid code is deprecated ?
• making smaller images to make pyramids from
  • needs changes in omero-py as the methods live there
  • possibly also omero-server would need to be reconfigured for lower "large image" limit, with undesirable consequences for other tests
• splitting the tests
  • still not sure how this will help and mainly how this is to be achieved. Note that OmeroPy tests are enough per se to create the race conditions and they do not "need" the preceding OmeroJava tests to create the unclean states, races, timeouts
  • the pyramid tests could be rather marked as local-only, which would be more logical (see above)

Handy setup for running OmeroPy suite only via Jenkins job:

1. Run omero-push and omero-build jobs
2. Check that snoopycrimecop repo has on "merge-ci" branch the recent changes (merged open PRs from openmicroscopy repo)
3. Edit the Config of the OMERO-test-integration job as follows

...
echo Running the integration tests with -Dtestng.useDefaultListeners=true

# commented out by PW 21 July 2026 to spped up the investigation of the OmeroPy suite
# $SRC/build.py -f components/tools/OmeroJava/build.xml -Dtestng.useDefaultListeners=true -Dtestreports.dir=target/reports/integration integration
$SRC/build.py -f components/tools/OmeroPy/build.xml integration  -Dtestreports.dir=target/reports/integration
# $SRC/build.py -f components/tools/OmeroFS/build.xml integration -Dtestreports.dir=target/reports/integration
$SRC/build.py -f components/tools/OmeroWeb/build.xml integration -Dtestreports.dir=target/reports/integration

## Broken
#$SRC/build.py -f components/tools/OmeroPy/build.xml integration -DMARK=broken -Dtestreports.dir=target/reports/broken
#$SRC/build.py -f components/tools/OmeroJava/build.xml -Dtestng.useDefaultListeners=true -Dtestreports.dir=target/reports/broken broken

deactivate

4. Run the OMERO-test-integration job - this will take just 2.5hours instead of ~6h.

cc @jburel @sbesson

Start with the Jenkins OMERO-test-integration report and the OmeroPy integration suite, especially tests using wait_for_pyramid, then compare the referenced wait_for_pyramid_file and renderability changes. Reproduce the NFS failures and determine whether pyramid readiness or OmeroWeb state races are responsible; done means the affected integration tests pass reliably with an agreed timeout or scope.