ENH: Compute hash of xarray objects
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
Is your feature request related to a problem? Please describe.
I'm working on some caching/data-provenance functionality for xarray objects, and I realized that there's no standard/efficient way of computing hashes for xarray objects.
Describe the solution you'd like
It would be useful to have a configurable, reliable/standard .hexdigest() method on xarray objects. For example, zarr provides a digest method that returns you a digest/hash of the data.
In [16]: import zarr
In [17]: z = zarr.zeros(shape=(10000, 10000), chunks=(1000, 1000))
In [18]: z.hexdigest() # uses sha1 by default for speed
Out[18]: '7162d416d26a68063b66ed1f30e0a866e4abed60'
In [20]: z.hexdigest(hashname='sha256')
Out[20]: '46fc6e52fc1384e37cead747075f55201667dd539e4e72d0f372eb45abdcb2aa'
I'm thinking that an xarray's built-in hashing mechanism would provide a more reliable way to treat metadata such as global attributes, encoding, etc... during the hash computation...
Describe alternatives you've considered
So far, I am using joblib's default hasher: joblib.hash() function. However, I am in favor of having a configurable/built-in hasher that is aware of xarray's data model and quirks :)
In [1]: import joblib
In [2]: import xarray as xr
In [3]: ds = xr.tutorial.open_dataset('rasm')
In [5]: joblib.hash(ds, hash_name='sha1')
Out[5]: '3e5e3f56daf81e9e04a94a3dff9fdca9638c36cf'
In [8]: ds.attrs = {}
In [9]: joblib.hash(ds, hash_name='sha1')
Out[9]: 'daab25fe735657e76514040608fadc67067d90a0'
Additional context
Add any other context about the feature request here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named in the issue. Start by reviewing xarray’s data model and existing handling of data, metadata, attributes, and encoding, then determine the hashing scope and configuration needed for a reliable hexdigest method. Done means the behavior is specified for xarray objects and covered by tests for data and metadata changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100