fsspec / fsspec/kerchunk

AsyncGroup.create_array() got an unexpected keyword argument 'compression' in netcdf3.py module

Open
#534 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
366
Forks
96
Avg merge
24m
Merged PRs (30d)
1

Description

I'm trying to create fsspec files for netCDF 3 datasets on a local filesystem, but am encountering an error during the json reference file creation process:

AsyncGroup.create_array() got an unexpected keyword argument 'compression'

The error is coming from the netCDF3.py module NetCDF3toZarr translate method in the following block:

     arr = z.create_dataset(
                    name=dim,
                    shape=shape,
                    dtype=var.data.dtype,
                    fill_value=fill,
                    chunks=shape,
                    compression=None,
                )

I'm using Kerchunk v0.2.7.

compression is not included in the create_dataset parameter list according to the zarr documentation, but compressor is, so fixing the argument in this and other create_dataset calls should presumably solve the issue.

Steps to reproduce:

def write_fsspec(fs_read, input_file, output_dir):
    with fs_read.open(input_file) as infile:
        print(f"Running kerchunk generation for {input_file}...")
        chunks = kerchunk.netCDF3.NetCDF3ToZarr(infile, inline_threshold=300)
        file_name = os.path.basename(input_file)
        file_name = file_name.replace('.nc', '.json')
        out_file_name = output_dir + '/' + file_name
        with open(out_file_name, "wb") as f:
            f.write(json.dumps(chunks.translate()).encode()) # call to netCDF3.py originates here
        print(f"Finished writing {out_file_name}")
        return out_file_name

 dir_path = config['input_dir']
 dir_path += '**/*.nc'
 file_paths = glob.glob(dir_path, recursive=True)
 fs_read = fsspec.filesystem('local')
 temp_dir = TemporaryDirectory(prefix=config['output_dir'])
 output_files = [write_fsspec(fs_read, f, temp_dir.name) for f in file_paths]

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in netCDF3.py at NetCDF3ToZarr.translate and the create_dataset block around lines 248-255. Run the provided local-filesystem reproduction, inspect the other create_dataset calls in that module, and verify that translating the netCDF3 dataset completes without the unexpected-keyword error and produces the JSON reference file.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.