pydata / pydata/xarray

ENH: Compute hash of xarray objects

Open
#4,738 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

Is your feature request related to a problem? Please describe.

I'm working on some caching/data-provenance functionality for xarray objects, and I realized that there's no standard/efficient way of computing hashes for xarray objects.

Describe the solution you'd like

It would be useful to have a configurable, reliable/standard .hexdigest() method on xarray objects. For example, zarr provides a digest method that returns you a digest/hash of the data.

In [16]: import zarr

In [17]: z = zarr.zeros(shape=(10000, 10000), chunks=(1000, 1000))

In [18]: z.hexdigest() # uses sha1 by default for speed
Out[18]: '7162d416d26a68063b66ed1f30e0a866e4abed60'

In [20]: z.hexdigest(hashname='sha256')
Out[20]: '46fc6e52fc1384e37cead747075f55201667dd539e4e72d0f372eb45abdcb2aa'

I'm thinking that an xarray's built-in hashing mechanism would provide a more reliable way to treat metadata such as global attributes, encoding, etc... during the hash computation...

Describe alternatives you've considered

So far, I am using joblib's default hasher: joblib.hash() function. However, I am in favor of having a configurable/built-in hasher that is aware of xarray's data model and quirks :)

In [1]: import joblib

In [2]: import xarray as xr

In [3]: ds = xr.tutorial.open_dataset('rasm')

In [5]: joblib.hash(ds, hash_name='sha1')
Out[5]: '3e5e3f56daf81e9e04a94a3dff9fdca9638c36cf'

In [8]: ds.attrs = {}

In [9]: joblib.hash(ds, hash_name='sha1')
Out[9]: 'daab25fe735657e76514040608fadc67067d90a0'

Additional context
Add any other context about the feature request here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named in the issue. Start by reviewing xarray’s data model and existing handling of data, metadata, attributes, and encoding, then determine the hashing scope and configuration needed for a reliable hexdigest method. Done means the behavior is specified for xarray objects and covered by tests for data and metadata changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.