RS_FromNetCDF fails on NetCDF4/HDF5 with filter type=32015 (e.g. ZSTD)
- Dominant language
- Java
- Stars
- 2.4k
- Forks
- 784
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 58
Description
I encountered a runtime failure when using RS_FromNetCDF(content, 'v') on a valid NetCDF4 file stored in S3.
The error is:
```Exception occurred while evaluating expression RS_FromNetCDF - inputs: [[B@xxxx, v], cause: Unknown filter type=32015```
This happens because the file uses an HDF5 compression filter (likely BLOSC, filter ID 32015), which is not supported by the version of netcdf-java (4.6.11) used in netcdfAll.
Steps to reproduce:
Load .nc file containing BLOSC/ZSTD compression into Spark via binaryFile.
Try to evaluate RS_FromNetCDF(content, 'variable').
A workaround is to re-encode the file using nccopy or xarray to remove unsupported filters, but this limitation is undocumented and leads to runtime crashes.
Expected:
An informative exception or warning.
Ideally: support for filter 32015
Environment:
Sedona: 1.7.1
Spark: 3.5.0
File: NetCDF4/HDF5 with ZSTD filter
Contributor guide
Research direction
Start at the RS_FromNetCDF entry point and trace how the netcdfAll 4.6.11 dependency handles NetCDF4/HDF5 filter type 32015. Reproduce with the reported compressed file or equivalent input, then verify that the result is either an informative warning or exception, or support for the filter rather than an unhelpful runtime crash.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100