huggingface / huggingface/candle
Unable to read older pytorch pickle files
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
It seems candle's pickle support currently assumes the newer pickle format torch switched to some time ago. (At least, I'm assuming this is the case). This is a slight issue as many of the .pth files I'd like to be able to load using candle are using this older format.
Specifically, I'm getting a `Zip(InvalidArchive("Could not find central directory end"))` when trying to use pickle::read_all on many older .pth files. Newer ones seem to work fine, but those are also unusable due to the load key issue I've mentioned previously.
My hunch is that this is due to torch switching how they pickle their path files. I remember at some point torch.save had an option to use the newer zip format, so I'm guessing these old ones are using something other than zip to store the pickle data, and thus the zip library can't read it properly. I'm not sure how to confirm this though.
I've added support for .pth reading to my [candle ESRGAN repo](https://github.com/joeyballentine/ESRGAN-candle-rs), where this can be pretty easily recreated. One of these model files should trigger it, as they are older files:
- https://openmodeldb.info/models/4x-BS-Deviance
- https://openmodeldb.info/models/4x-FSDedither
- https://openmodeldb.info/models/4x-DigiPaint
- https://openmodeldb.info/models/4x-Manga109Attempt
If you need a smaller repro, I can try to provide one when I have time tomorrow.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with pickle::read_all and reproduce the Zip(InvalidArchive("Could not find central directory end")) error using one of the older .pth files linked in the issue or the candle ESRGAN repository. Compare the older files with newer files that already load, and consider the work done when the listed older models can be read without breaking newer .pth support.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pytorch, rust
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100