NCAS-CMS / NCAS-CMS/cf-python

Error in cf.write when writing UM data fields

Open
#838 12 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question um/pp
Dominant language
Python
Stars
150
Forks
23
Avg merge
1d 11h
Merged PRs (30d)
2

Description

Hi folks. I'm encountering issues again with processing UM data. It's a related task to ticket #817 but a different error so I figured I'd start a new topic.

I can load in a single file (either on a surface level, UM tiles, or pressure levels) with no problem and perform usual cf-python tasks like amending and adding attributes, but when it comes to writing the file I get an unhashable type error as follows:

>>> cf.write(f[0], 'test.nc')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.11/mf3-23.11.0-0/envs/jaspy3.11-mf3-23.11.0-0-v20240815/lib/python3.11/site-packages/cfdm/decorators.py", line 171, in verbose_override_wrapper
    return method_with_verbose_kwarg(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.11/mf3-23.11.0-0/envs/jaspy3.11-mf3-23.11.0-0-v20240815/lib/python3.11/site-packages/cf/read_write/write.py", line 808, in write
    netcdf.write(
  File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.11/mf3-23.11.0-0/envs/jaspy3.11-mf3-23.11.0-0-v20240815/lib/python3.11/site-packages/cfdm/decorators.py", line 171, in verbose_override_wrapper
    return method_with_verbose_kwarg(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.11/mf3-23.11.0-0/envs/jaspy3.11-mf3-23.11.0-0-v20240815/lib/python3.11/site-packages/cfdm/read_write/netcdf/netcdfwrite.py", line 4911, in write
    self._file_io_iteration(
  File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.11/mf3-23.11.0-0/envs/jaspy3.11-mf3-23.11.0-0-v20240815/lib/python3.11/site-packages/cfdm/read_write/netcdf/netcdfwrite.py", line 5153, in _file_io_iteration
    self._write_global_attributes(fields)
  File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.11/mf3-23.11.0-0/envs/jaspy3.11-mf3-23.11.0-0-v20240815/lib/python3.11/site-packages/cfdm/read_write/netcdf/netcdfwrite.py", line 4289, in _write_global_attributes
    force_global = {
                   ^
  File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.11/mf3-23.11.0-0/envs/jaspy3.11-mf3-23.11.0-0-v20240815/lib/python3.11/site-packages/cfdm/read_write/netcdf/netcdfwrite.py", line 4292, in <dictcomp>
    if len(v) == len(fields) and len(set(v)) == 1
                                     ^^^^^^
TypeError: unhashable type: 'dict'

I've managed to process hourly and daily files with this workflow and v similar code with no problem so far. I've also double-checked that all the files I'm currently trying to process here are fully formed and compliant, so that's not the issue (I think this was the main problem in my previous ticket).

I have also tested this behaviour with a MWE, where I load a UM file and then try to write it to netcdf, and the same occurs, i.e.

f = cf.read('/path/to/file/UM_file')
cf.write(f[0], 'test.nc')

Can you help me understand what's going on here? I am able to save the fields in iris without a problem, so it doesn't seem to be packing or underlying file problem, but I'd like to use cf-python to maintain consistency with the other files I've post-processed with this method.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with the UM file and minimal cf.read/cf.write example. Start at cf/read_write/write.py and the cfdm netcdfwrite.py _write_global_attributes path named in the traceback, then determine why the global-attribute values contain an unhashable dict. Done means the UM field writes successfully without this TypeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.