Error in read_record_data when processing 6-hourly UM data
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 150
- Forks
- 23
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 2
Description
I'm getting an error when trying to collapse UM files to go from sub-daily to daily frequency. The data are 6-hourly maximum air temperature (max across all timesteps).
I'm new to cf-python so unsure about where/how to start with debugging this, and read_record_data doesn't tell me much about the source of the error:
umfile: error condition detected in routine read_record_data_core_dbl
umfile: error condition detected in routine read_record_data
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.11/mf3-23.11.0-0/envs/jaspy3.11-mf3-23.11.0-0-v20240815/lib/python3.11/site-packages/cf/decorators.py", line 71, in precede_with_kwarg_deprecation_check
operation_method_result = operation_method(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.11/mf3-23.11.0-0/envs/jaspy3.11-mf3-23.11.0-0-v20240815/lib/python3.11/site-packages/cfdm/decorators.py", line 171, in verbose_override_wrapper
return method_with_verbose_kwarg(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.11/mf3-23.11.0-0/envs/jaspy3.11-mf3-23.11.0-0-v20240815/lib/python3.11/site-packages/cf/field.py", line 6931, in collapse
f = f._collapse_grouped(
^^^^^^^^^^^^^^^^^^^^
File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.11/mf3-23.11.0-0/envs/jaspy3.11-mf3-23.11.0-0-v20240815/lib/python3.11/site-packages/cfdm/decorators.py", line 171, in verbose_override_wrapper
return method_with_verbose_kwarg(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.11/mf3-23.11.0-0/envs/jaspy3.11-mf3-23.11.0-0-v20240815/lib/python3.11/site-packages/cf/field.py", line 8628, in _collapse_grouped
f = self.concatenate(fl, axis=iaxis, cull_graph=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.11/mf3-23.11.0-0/envs/jaspy3.11-mf3-23.11.0-0-v20240815/lib/python3.11/site-packages/cf/field.py", line 3189, in concatenate
new_data = Data.concatenate(
^^^^^^^^^^^^^^^^^
File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.11/mf3-23.11.0-0/envs/jaspy3.11-mf3-23.11.0-0-v20240815/lib/python3.11/site-packages/cf/data/data.py", line 4068, in concatenate
d.cull_graph()
File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.11/mf3-23.11.0-0/envs/jaspy3.11-mf3-23.11.0-0-v20240815/lib/python3.11/site-packages/cf/data/data.py", line 11470, in cull_graph
dsk, _ = cull(dx.dask, dx.__dask_keys__())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.11/mf3-23.11.0-0/envs/jaspy3.11-mf3-23.11.0-0-v20240815/lib/python3.11/site-packages/dask/optimization.py", line 61, in cull
dependencies_k = get_dependencies(dsk, k, as_list=True) # fuse needs lists
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.11/mf3-23.11.0-0/envs/jaspy3.11-mf3-23.11.0-0-v20240815/lib/python3.11/site-packages/dask/core.py", line 306, in get_dependencies
return keys_in_tasks(dsk, [arg], as_list=as_list)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.11/mf3-23.11.0-0/envs/jaspy3.11-mf3-23.11.0-0-v20240815/lib/python3.11/site-packages/dask/core.py", line 194, in keys_in_tasks
if w in keys:
^^^^^^^^^
File "<frozen _collections_abc>", line 780, in __contains__
File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.11/mf3-23.11.0-0/envs/jaspy3.11-mf3-23.11.0-0-v20240815/lib/python3.11/site-packages/dask/highlevelgraph.py", line 517, in __getitem__
return self.layers[key[0]][key] # type: ignore
~~~^^^
File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.11/mf3-23.11.0-0/envs/jaspy3.11-mf3-23.11.0-0-v20240815/lib/python3.11/site-packages/cf/data/array/umarray.py", line 190, in __getitem__
array = rec.get_data().reshape(self.shape)
^^^^^^^^^^^^^^
File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.11/mf3-23.11.0-0/envs/jaspy3.11-mf3-23.11.0-0-v20240815/lib/python3.11/site-packages/cf/umread_lib/umfile.py", line 435, in get_data
return c.read_record_data(
^^^^^^^^^^^^^^^^^^^
File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.11/mf3-23.11.0-0/envs/jaspy3.11-mf3-23.11.0-0-v20240815/lib/python3.11/site-packages/cf/umread_lib/cInterface.py", line 591, in read_record_data
raise umfile.UMFileException("Error reading record data")
cf.umread_lib.umfile.UMFileException: Error reading record data
Any help would be appreciated!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the 6-hourly UM collapse to daily frequency described in the issue. Read cf/field.py around collapse and concatenate, then follow the traceback through cf/data/array/umarray.py, cf/umread_lib/umfile.py, and cf/umread_lib/cInterface.py. Done means identifying the cause of the record-reading failure and confirming that the collapse completes successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100