Counting broken on single machines (non-MPI)

Counting using the docker containers on a single machine is not working correctly. This was recently discovered when the docker image on mothership6 was upgraded from stempy 3.3.5 to stempy:latest (currently 3.3.16).

On mothership6, we use singularity which is similar to docker. We run a counting job as daquser using a script (~/4dc_procesing/count.sh). Here is a simplified one-liner:
$ /usr/local/bin/singularity exec -B /mnt /home/daquser/stempy_latest.sif python3 ~/4dc_processing/electron_count.py -t 4.5 --out-dir /mnt/hdd2/ -s 28 -v
-s <scan number> is how you change the scan number (see below).

Proper counting is done when using /home/daquser/stempy_3.3.5.sif (3.3.6 and 3.3.7 work too). Incorrect counting is done with /home/daquser/stempy_latest.sif or stempy_py39-0188853.sif

The difference in the counted files is as follows:

stempy 3.3.5:

$ /usr/local/bin/singularity exec -B /mnt /home/daquser/stempy_3.3.5.sif python3 ~/4dc_processing/electron_count.py -t 4.5 --out-dir /mnt/hdd2/ -s 23

>>> import h5py
>>> f0 = h5py.File('/mnt/hdd2/data_scan23_id0000_electrons.h5', 'r')
>>> f0['electron_events/frames']
<HDF5 dataset "frames": shape (16512,), type "|O">
>>> f0['electron_events/frames'][0]
array([136834, 184903, 260178], dtype=uint32)

stempy 3.3.16 (latest)

/usr/local/bin/singularity exec -B /mnt /home/daquser/stempy_latest.sif python3 ~/4dc_processing/electron_count.py -t 4.5 --out-dir /mnt/hdd2/ -s 23

>>> import h5py
>>> f1 = h5py.File('/mnt/hdd2/data_scan23_id0000_electrons_001.h5', 'r')
>>> f1['electron_events/frames'][0]
array([136834, 136834, 136834], dtype=uint32)

Note that the first frame only has 3 electron hits for this data set. The data counted with stempy v3.3.5 has three different electron event locations. The one counted with the newer stempy (unsure of exact version) has the same electron event three times (which is impossible for this dataset).

Version testing

I tested stempy_py39-fc4253c.sif (stempy 3.3.6) and stempy_py39-2c8aa53.sif (3.3.7). Both of these produced good files.

Then I tested with stempy_py39-0188853.sif which is the same as stempy_latest.sif and I see the problem. Further version testing needed.

NERSC

Please note that counting on NERSC works. See distiller job 9631 and scan id 21901 from 2024.11.13. Counting worked as expected:

import h5py
f0 = h5py.File('/global/cfs/cdirs/ncemhub/distiller/counted/2024.11.13/FOURD_241113_1047_21901_00023.h5')

f0['/electron_events/frames'][0]
array([136834, 184903, 260178], dtype=uint32)

Start by reproducing the issue through electron_count.py in the listed Singularity images, comparing stempy 3.3.5–3.3.7 with stempy_py39-0188853.sif and stempy_latest.sif. Inspect the generated HDF5 electron_events/frames values with h5py; done means identifying the regression and restoring distinct electron event locations for non-MPI single-machine counting.