apache / apache/sedona

RS_FromNetCDF fails on NetCDF4/HDF5 with filter type=32015 (e.g. ZSTD)

Open
#1,951 2 comments 0 reactions 0 assignees View on GitHub
good first issue
Dominant language
Java
Stars
2.4k
Forks
784
Avg merge
1d 12h
Merged PRs (30d)
58

Description

I encountered a runtime failure when using RS_FromNetCDF(content, 'v') on a valid NetCDF4 file stored in S3.

The error is:

```Exception occurred while evaluating expression RS_FromNetCDF - inputs: [[B@xxxx, v], cause: Unknown filter type=32015```

This happens because the file uses an HDF5 compression filter (likely BLOSC, filter ID 32015), which is not supported by the version of netcdf-java (4.6.11) used in netcdfAll.

Steps to reproduce:

Load .nc file containing BLOSC/ZSTD compression into Spark via binaryFile.

Try to evaluate RS_FromNetCDF(content, 'variable').

A workaround is to re-encode the file using nccopy or xarray to remove unsupported filters, but this limitation is undocumented and leads to runtime crashes.

Expected:

An informative exception or warning.

Ideally: support for filter 32015

Environment:

Sedona: 1.7.1

Spark: 3.5.0

File: NetCDF4/HDF5 with ZSTD filter

Contributor guide

Open the contributing guide

Research direction

Start at the RS_FromNetCDF entry point and trace how the netcdfAll 4.6.11 dependency handles NetCDF4/HDF5 filter type 32015. Reproduce with the reported compressed file or equivalent input, then verify that the result is either an informative warning or exception, or support for the filter rather than an unhelpful runtime crash.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.