EleutherAI / EleutherAI/sparsify

micro_acc_steps flag functionality

Open
#111 1 comment 0 reactions 1 assignee Claimed by @KartikAhluwalia View on GitHub
bug good first issue
Dominant language
Python
Stars
739
Forks
105
PR merge metrics
No merged PRs in 30d

Description

### `micro_acc_steps`: the documentation says the flag implements microbatching, but there seems no such functionality.
#### Expected behaviour (from README `--distribute_modules` example)

> “It accumulates gradients over **8** minibatches, **and splits each minibatch into 2 microbatches** before feeding them into the SAE encoder, thus saving a lot of memory.”
> ```bash
> torchrun … --grad_acc_steps 8 … **--micro_acc_steps 2**
> ```

---

#### Actual behaviour in the code

- `sparsify/config.py`:
```python
micro_acc_steps: int = 1 # "Chunk the activations into this number of microbatches for training"
```
- `sparsify/trainer.py` (**only place** the value is used):
```python
acc_steps = self.cfg.grad_acc_steps * self.cfg.micro_acc_steps
```

I don't see actual split on the `micro_acc_steps` minibatches, and the activations are fed to the SAE whole, regardless of the `micro_acc_steps` value.

---

From what I can see, setting `micro_acc_steps > 1` only multiplies the gradient-accumulation denominator (`acc_steps`). That means the effective learning rate goes down, but the memory footprint stays the same.

If that’s correct, it might be worth updating the README (and the flag’s doc-string in `config.py`) to avoid confusion for new users.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.