bmaltais / bmaltais/dehydrate

Compression using sparse tensors and thresholding

Open
#1 11 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
52
Forks
12
PR merge metrics
No merged PRs in 30d

Description

Not sure what the original script did since it had a default alpha value of 0.

```py
parser.add_argument("--str", type=float, help="Strength of the rehydration (-0.05..0.05)", default=0, required=False)`
alpha = args.str
...
for key in tqdm(theta_0.keys(), desc="Stage 1/2?: merge common keys"):
if "model" in key and key in theta_1:
theta_0[key] = theta_0[key] * (1 + alpha) + theta_1[key] * (1 - alpha)
```

Which appears just replaced the final weights with weights that were the same, since `theta_0[key] = theta_0[key] * 1 + theta_1[key] * 0` = `theta_0[key] = theta_0[key]`.

Constructing a patch with M1 - M2 = patch and then M1 + patch = M2 does not work due to lossy FP and seems to give me corrupted models.

What I did instead was use thresholding based on the min/max delta of the individual tensors and then apply the most changed values as a patch.

Here's my code: https://gist.github.com/AmericanPresidentJimmyCarter/1947162f371e601ce183070443f41dc2

This results in an about 60 MB patch, but the quality of transfer is not great. You can increase the THRESHOLD_STRENGTH variable but the file sizes increase dramatically while the quality does not seem to. Here are results at `THRESHOLD_STRENGTH == 2.`.

![elon_disney_patch](https://user-images.githubusercontent.com/110263573/198841767-1659bd73-9dbd-48ef-a5b1-fe983752da09.jpg)

Test code is:

```py
_, extra_data = engine.sample(
'elon musk in modern disney style',
4,
'heun',
679566949,
25,
scale=7.,
)
```

Using dreambooth'd weights here: https://huggingface.co/nitrosocke/mo-di-diffusion

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the original script excerpt and the linked gist, then reproduce the issue with the provided engine.sample call and the nitrosocke/mo-di-diffusion weights. Compare the thresholding approach's patch size and transfer quality across threshold strengths; done means a documented compression method that avoids corrupted models and has acceptable quality.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.