NCAS-CMS / NCAS-CMS/cf-python

error from netCDF in cf.write

Open
#960 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
150
Forks
23
Avg merge
1d 11h
Merged PRs (30d)
2

Description

I am processing some CMIP6 data on Jasmin (making time-means). I get the following error. I realise that you probably need more background to diagnose it, but perhaps it's obvious from this much information? As a result of the error, the file is not properly written, just the start of it.

Thanks. Jonathan

Traceback (most recent call last):
  File "/home/users/jonathan/bin/grouptime", line 97, in <module>
    cf.write(n,ofile)
  File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/cfdm/read_write/write.py", line 826, in __new__
    netcdf.write(
  File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/cfdm/decorators.py", line 171, in verbose_override_wrapper
    return method_with_verbose_kwarg(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/cfdm/read_write/netcdf/netcdfwrite.py", line 5232, in write
    self._file_io_iteration(
  File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/cfdm/read_write/netcdf/netcdfwrite.py", line 5478, in _file_io_iteration
    self._write_global_attributes(fields)
  File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/cfdm/read_write/netcdf/netcdfwrite.py", line 4610, in _write_global_attributes
    g["netcdf"].setncattr(
  File "src/netCDF4/_netCDF4.pyx", line 3087, in netCDF4._netCDF4.Dataset.setncattr
  File "src/netCDF4/_netCDF4.pyx", line 1858, in netCDF4._netCDF4._set_att
  File "src/netCDF4/_netCDF4.pyx", line 6733, in netCDF4._netCDF4._strencode
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 61-63: surrogates not allowed

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with cfdm/read_write/netcdf/netcdfwrite.py at _write_global_attributes and the netCDF4 Dataset.setncattr call shown in the traceback. Reproduce cf.write in the reported Python environment and identify which global attribute contains the invalid surrogate characters. Done means the affected data can be written without the UnicodeEncodeError and the behavior is covered by an appropriate regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.