DTM or DSM jobs failing

webodm 3.0.1 nodeODM:latest
FastOrtho job finishes successfully
any DTM or DSM fail with
ImportError: cannot import name '_gdal_array' from 'osgeo' (/code/venv/lib/python3.12/site-packages/osgeo/__init__.py)

[INFO]    Computing raster stats for /var/www/data/6fa96244-8b2a-41a8-ab1b-9da86fb6ae07/odm_dem/dsm.preview.tif
[INFO]    running gdaldem color-relief "/var/www/data/6fa96244-8b2a-41a8-ab1b-9da86fb6ae07/odm_dem/dsm.preview.tif" "/code/opendm/tiles/color_relief.txt" "/var/www/data/6fa96244-8b2a-41a8-ab1b-9da86fb6ae07/odm_dem/dsm.previewcolor.tif" -alpha -co ALPHA=YES
0...10...20...30...40...50...60...70...80...90...100 - done.
[INFO]    running gdaldem hillshade "/var/www/data/6fa96244-8b2a-41a8-ab1b-9da86fb6ae07/odm_dem/dsm.preview.tif" "/var/www/data/6fa96244-8b2a-41a8-ab1b-9da86fb6ae07/odm_dem/dsm.previewhillshade.tif" -z 1.0 -s 1.0 -az 315.0 -alt 45.0
0...10...20...30...40...50...60...70...80...90...100 - done.
[INFO]    running "/code/venv/bin/python3" "/code/opendm/tiles/hsv_merge.py" "/var/www/data/6fa96244-8b2a-41a8-ab1b-9da86fb6ae07/odm_dem/dsm.previewcolor.tif" "/var/www/data/6fa96244-8b2a-41a8-ab1b-9da86fb6ae07/odm_dem/dsm.previewhillshade.tif" "/var/www/data/6fa96244-8b2a-41a8-ab1b-9da86fb6ae07/odm_dem/dsm.previewcolored_hillshade.tif"
/code/venv/lib/python3.12/site-packages/osgeo/gdal.py:312: FutureWarning: Neither gdal.UseExceptions() nor gdal.DontUseExceptions() has been explicitly called. In GDAL 4.0, exceptions will be enabled by default.
warnings.warn(
Traceback (most recent call last):
File "/code/opendm/tiles/hsv_merge.py", line 198, in <module>
rScanline = rBand.ReadAsArray(0, i, hillband.XSize, 1, hillband.XSize, 1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/code/venv/lib/python3.12/site-packages/osgeo/gdal.py", line 5283, in ReadAsArray
from osgeo import gdal_array
File "/code/venv/lib/python3.12/site-packages/osgeo/gdal_array.py", line 10, in <module>
from . import _gdal_array
ImportError: cannot import name '_gdal_array' from 'osgeo' (/code/venv/lib/python3.12/site-packages/osgeo/__init__.py)
[WARNING] Cannot generate colored hillshade: Child returned 1
[INFO]    running gdal_translate -outsize 1400 0 -of png "None" "/var/www/data/6fa96244-8b2a-41a8-ab1b-9da86fb6ae07/opensfm/stats/dsm.png" --config GDAL_CACHEMAX 32.75%
ERROR 4: None: No such file or directory

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
File "/code/stages/odm_app.py", line 82, in execute
self.first_stage.run()
File "/code/opendm/types.py", line 470, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 470, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 470, in run
self.next_stage.run(outputs)
[Previous line repeated 8 more times]
File "/code/opendm/types.py", line 449, in run
self.process(self.args, outputs)
File "/code/stages/odm_report.py", line 214, in process
system.run("gdal_translate -outsize {} 0 -of png \"{}\" \"{}\" --config GDAL_CACHEMAX {}%".format(image_target_size, colored_hillshade_dem, osfm_dem, get_max_memory()))
File "/code/opendm/system.py", line 112, in run
raise SubprocessException("Child returned {}".format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1

===== Done, human-readable information to follow... =====

[ERROR]   Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See https://docs.opendronemap.org/flying/
100 - done.

guided by perplexity, i tried all this inside the docker container for nodeodm:

# gdalinfo --version
GDAL 3.8.4, released 2024/02/08

# python3 -c "import osgeo; print(osgeo.__version__)"
3.8.4

# pip list | grep -i gdal
GDAL               3.8.4

# python3 -c "from osgeo import _gdal_array; print('OK')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: cannot import name '_gdal_array' from 'osgeo' (/code/venv/lib/python3.12/site-packages/osgeo/__init__.py)

# pip install --upgrade pip setuptools wheel
... (boring results redacted)

# pip install GDAL==3.8.4 --only-binary=GDAL
Requirement already satisfied: GDAL==3.8.4 in /code/venv/lib/python3.12/site-packages (3.8.4)

# python3 -c "from osgeo import _gdal_array; print('OK')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: cannot import name '_gdal_array' from 'osgeo' (/code/venv/lib/python3.12/site-packages/osgeo/__init__.py)

reverting to nodeODM:stable seems to fix it

Start with opendm/tiles/hsv_merge.py at line 198, where ReadAsArray imports gdal_array, and inspect the nodeODM:latest environment described in the report. Compare it with nodeODM:stable and verify the GDAL Python package contents. Done means DTM and DSM jobs complete without the _gdal_array ImportError and generate their expected raster outputs.