huggingface / huggingface/datasets
record_checksums default changed True -> False in 4.8.5, not in the release notes
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### What changed
Between 4.8.4 and 4.8.5, checksum recording flipped from on to off by default in two places. From the sdists:
```
4.8.4 src/datasets/download/download_manager.py:80 record_checksums=True
4.8.5 src/datasets/download/download_manager.py:80 record_checksums=False
4.8.4 src/datasets/utils/info_utils.py:80 record_checksum: bool = True
4.8.5 src/datasets/utils/info_utils.py:80 record_checksum: bool = False
```
### Why I am asking rather than reporting
The 4.8.5 release notes do not contain the words checksum, integrity, or verification. The closest entry is:
> Don't include files list DatasetInfo (and remove old stuff) by @lhoestq in #8128
whose stated motivation is that `DatasetInfo` was making the viewer's Mongo documents too large, with checksum recording described as cleanup alongside it. So this reads to me like a deliberate change that came in as a side effect of an unrelated fix, rather than a decision anyone announced.
I may well be misreading it, which is why this is a question.
### The reason it matters to someone downstream
`DatasetInfo.download_checksums` stops being populated, so anything relying on it for provenance or reproducibility silently gets nothing rather than an error. It went out in a patch release, where the version number promises no behaviour change at all.
### What would help
Either a line in the release notes or a note in the docs confirming that checksum recording is now opt-in, and ideally a sentence on what the recommended replacement is for verifying downloaded files. If it was not intended to change the default, that is worth knowing too.
### How I found it
A tool I maintain diffs default values of public callables between two releases and grades by whether the version number promised anything. This came out of a sweep across 1,253 consecutive patch releases; it was one of a small number where the changed default looked consequential.
Contributor guide
Research direction
Start with src/datasets/download/download_manager.py and src/datasets/utils/info_utils.py at the cited record_checksums and record_checksum defaults, then review the 4.8.5 release notes and the change from #8128. Confirm whether the default change was intentional and document it, including the recommended verification approach, in the release notes or relevant docs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100