huggingface / huggingface/candle

Unable to read older pytorch pickle files

Open
#1,348 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

It seems candle's pickle support currently assumes the newer pickle format torch switched to some time ago. (At least, I'm assuming this is the case). This is a slight issue as many of the .pth files I'd like to be able to load using candle are using this older format.

Specifically, I'm getting a `Zip(InvalidArchive("Could not find central directory end"))` when trying to use pickle::read_all on many older .pth files. Newer ones seem to work fine, but those are also unusable due to the load key issue I've mentioned previously.

My hunch is that this is due to torch switching how they pickle their path files. I remember at some point torch.save had an option to use the newer zip format, so I'm guessing these old ones are using something other than zip to store the pickle data, and thus the zip library can't read it properly. I'm not sure how to confirm this though.

I've added support for .pth reading to my [candle ESRGAN repo](https://github.com/joeyballentine/ESRGAN-candle-rs), where this can be pretty easily recreated. One of these model files should trigger it, as they are older files:
- https://openmodeldb.info/models/4x-BS-Deviance
- https://openmodeldb.info/models/4x-FSDedither
- https://openmodeldb.info/models/4x-DigiPaint
- https://openmodeldb.info/models/4x-Manga109Attempt

If you need a smaller repro, I can try to provide one when I have time tomorrow.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with pickle::read_all and reproduce the Zip(InvalidArchive("Could not find central directory end")) error using one of the older .pth files linked in the issue or the candle ESRGAN repository. Compare the older files with newer files that already load, and consider the work done when the listed older models can be read without breaking newer .pth support.

Written by the indexing model from the issue text.

Assessment

Tech stack
pytorch, rust
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.