NCAS-CMS / NCAS-CMS/cf-python

Converting from kerchunk to CFA: fix or document behaviour

Open
#945 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

aggregation kerchunk
Dominant language
Python
Stars
150
Forks
23
Avg merge
1d 11h
Merged PRs (30d)
2

Description

Regarding our read/write formats support, we yesterday realised the case of going from (newly-supported in v3.20!) kerchunk to CFA, i.e. reading kerchunk and attempting to write it out as CFA, may not work on paper - and indeed when tetsing this I get a TypeError: expected string or bytes-like object, got 'FSMap' error, for example by adding a few lines to a relevant test to request a CFA-format write of read-in kerchunk, namely:

diff --git a/cf/test/test_kerchunk.py b/cf/test/test_kerchunk.py
index b37d09718d..98a9b89968 100644
--- a/cf/test/test_kerchunk.py
+++ b/cf/test/test_kerchunk.py
@@ -75,7 +75,10 @@ class read_writeTest(unittest.TestCase):
 
         fs = fsspec.filesystem("reference", fo=d)
         kerchunk = fs.get_mapper()
-        self.assertEqual(len(cf.read(kerchunk)), 1)
+        k = cf.read(kerchunk)
+        self.assertEqual(len(k), 1)
+        print("Attempting kerchunk -> CFA")
+        cf.write(k, "kerchunk_to_cfa.cfa", cfa="field")
 
     def test_read_bytes(self):
         """Test cf.read with a Kerchunk raw bytes representation."""

I see:

======================================================================
ERROR: test_read_dict (__main__.read_writeTest.test_read_dict)
Test cf.read with an Kerchunk dictionary.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/slb93/git-repos/cf-python/cf/test/test_kerchunk.py", line 81, in test_read_dict
    cf.write(k, "kerchunk_to_cfa.cfa", cfa="field")
  File "/home/slb93/git-repos/cfdm/cfdm/read_write/write.py", line 936, in __new__
    netcdf.write(
  File "/home/slb93/git-repos/cfdm/cfdm/decorators.py", line 171, in verbose_override_wrapper
    return method_with_verbose_kwarg(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/slb93/git-repos/cfdm/cfdm/read_write/netcdf/netcdfwrite.py", line 5952, in write
    self._file_io_iteration(
  File "/home/slb93/git-repos/cfdm/cfdm/read_write/netcdf/netcdfwrite.py", line 6230, in _file_io_iteration
    self._write_field_or_domain(f)
  File "/home/slb93/git-repos/cfdm/cfdm/read_write/netcdf/netcdfwrite.py", line 4746, in _write_field_or_domain
    self._write_netcdf_variable(
  File "/home/slb93/git-repos/cfdm/cfdm/read_write/netcdf/netcdfwrite.py", line 3310, in _write_netcdf_variable
    cfa = self._cfa_fragment_array_variables(data, cfvar)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/slb93/git-repos/cfdm/cfdm/read_write/netcdf/netcdfwrite.py", line 7014, in _cfa_fragment_array_variables
    uri = urisplit(dataset_name)
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/slb93/miniconda3/envs/cf-env-312-numpy2/lib/python3.12/site-packages/uritools/__init__.py", line 545, in urisplit
    return result(*result.RE.match(uristring).groups())
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: expected string or bytes-like object, got 'FSMap'

so there's at least an immediate logical issue to work through relating to type expectations for this case, but there we should work out whether the case can be supported - if so how much further work/logic is needed - and if not feasible to support soon we should document that this case doesn't (yet) work.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with cf/test/test_kerchunk.py and reproduce the CFA write through cf.write using the kerchunk object. Trace the failure from cfdm/read_write/netcdf/netcdfwrite.py, especially _cfa_fragment_array_variables and its urisplit call, to determine whether FSMap input can be supported. Done means a passing conversion test, or documentation that this conversion is unsupported.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.