Compression using sparse tensors and thresholding
- Dominant language
- Python
- Stars
- 52
- Forks
- 12
- PR merge metrics
- No merged PRs in 30d
Description
Not sure what the original script did since it had a default alpha value of 0.
```py
parser.add_argument("--str", type=float, help="Strength of the rehydration (-0.05..0.05)", default=0, required=False)`
alpha = args.str
...
for key in tqdm(theta_0.keys(), desc="Stage 1/2?: merge common keys"):
if "model" in key and key in theta_1:
theta_0[key] = theta_0[key] * (1 + alpha) + theta_1[key] * (1 - alpha)
```
Which appears just replaced the final weights with weights that were the same, since `theta_0[key] = theta_0[key] * 1 + theta_1[key] * 0` = `theta_0[key] = theta_0[key]`.
Constructing a patch with M1 - M2 = patch and then M1 + patch = M2 does not work due to lossy FP and seems to give me corrupted models.
What I did instead was use thresholding based on the min/max delta of the individual tensors and then apply the most changed values as a patch.
Here's my code: https://gist.github.com/AmericanPresidentJimmyCarter/1947162f371e601ce183070443f41dc2
This results in an about 60 MB patch, but the quality of transfer is not great. You can increase the THRESHOLD_STRENGTH variable but the file sizes increase dramatically while the quality does not seem to. Here are results at `THRESHOLD_STRENGTH == 2.`.

Test code is:
```py
_, extra_data = engine.sample(
'elon musk in modern disney style',
4,
'heun',
679566949,
25,
scale=7.,
)
```
Using dreambooth'd weights here: https://huggingface.co/nitrosocke/mo-di-diffusion
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the original script excerpt and the linked gist, then reproduce the issue with the provided engine.sample call and the nitrosocke/mo-di-diffusion weights. Compare the thresholding approach's patch size and transfer quality across threshold strengths; done means a documented compression method that avoids corrupted models and has acceptable quality.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100