AsyncGroup.create_array() got an unexpected keyword argument 'compression' in netcdf3.py module
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
I'm trying to create fsspec files for netCDF 3 datasets on a local filesystem, but am encountering an error during the json reference file creation process:
AsyncGroup.create_array() got an unexpected keyword argument 'compression'
The error is coming from the netCDF3.py module NetCDF3toZarr translate method in the following block:
arr = z.create_dataset(
name=dim,
shape=shape,
dtype=var.data.dtype,
fill_value=fill,
chunks=shape,
compression=None,
)
I'm using Kerchunk v0.2.7.
compression is not included in the create_dataset parameter list according to the zarr documentation, but compressor is, so fixing the argument in this and other create_dataset calls should presumably solve the issue.
Steps to reproduce:
def write_fsspec(fs_read, input_file, output_dir):
with fs_read.open(input_file) as infile:
print(f"Running kerchunk generation for {input_file}...")
chunks = kerchunk.netCDF3.NetCDF3ToZarr(infile, inline_threshold=300)
file_name = os.path.basename(input_file)
file_name = file_name.replace('.nc', '.json')
out_file_name = output_dir + '/' + file_name
with open(out_file_name, "wb") as f:
f.write(json.dumps(chunks.translate()).encode()) # call to netCDF3.py originates here
print(f"Finished writing {out_file_name}")
return out_file_name
dir_path = config['input_dir']
dir_path += '**/*.nc'
file_paths = glob.glob(dir_path, recursive=True)
fs_read = fsspec.filesystem('local')
temp_dir = TemporaryDirectory(prefix=config['output_dir'])
output_files = [write_fsspec(fs_read, f, temp_dir.name) for f in file_paths]
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in netCDF3.py at NetCDF3ToZarr.translate and the create_dataset block around lines 248-255. Run the provided local-filesystem reproduction, inspect the other create_dataset calls in that module, and verify that translating the netCDF3 dataset completes without the unexpected-keyword error and produces the JSON reference file.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100