LearningRate-Free Learning Algorithm

Hi, how about [D-adaptation](https://github.com/facebookresearch/dadaptation)?

This is a kind of algorithm that end-user doesn't need to set specific learning rate.

In short, D-adaptation use boundedness to find proper learning rate.

So, it might be useful to someone who hard to find hyperparameters.

Before I wrote this issue, I implement D-adaptation optimizer(Adam) for LoRA. It works!

A few code need to implementation. But I don't know all about sd-scripts code, there exists hard codings.

Requirement for D-dataptation is only torch>=1.5.1 and pip install dadaptation.

Here are codes.

In train_network.py
`
from torch.optim as optim # using for a raw learning rate scheduler
`
`
import dadaptation
`
and I hard-coded for applying optimizer.
`
optimizer = optimizer_class(trainable_params, lr=args.learning_rate)
`
to
`
optimizer = dadaptation.DAdaptAdam(trainable_params, lr=1.0, decouple=True, weight_decay=1.0)
`
Setting decople=True means that optimizer is AdamW not Adam. and weight_decay is for l2 penalty.

Other argumentation is not for end-user.(maybe)

And trainable_params doesn't need a specific learning rate, so replace
`
trainable_params = network.prepare_optimizer_params(args.text_encoder_lr, args.unet_lr)
`
to
`
trainable_params = network.prepare_optimizer_params(None, None)
`

In sd-scripts, lr_scheduler is a return of get_scheduler_fix function.

But I don't know why using get_scheduler_fix interrupt D-adaptation,

so I override lr_scheduler to LambdaLR. sorry for hard coding again :)

`
lr_scheduler = optim.lr_scheduler.LambdaLR(optimizer=optimizer, lr_lambda=[lambda epoch: 1, lambda epoch: 1],
last_epoch=-1,
verbose=False)
`

For monitoring dlr value,

`
logs['lr/d*lr'] = optimizer.param_groups[0]['d']*optimizer.param_groups[0]['lr']
`

might be needed. All things done.


_This image is d*lr-step graph when I use D-dadaptation._

I trained LoRA using D-adaptation, result is [here](https://civitai.com/models/7607/manhattan-cafe-umamusume).

Thank you!

Start in train_network.py, then trace get_scheduler_fix and the optimizer setup to understand how learning rates and scheduler state are connected. Compare the proposed D-adaptation integration with the existing training flow and determine how it should be exposed without hard-coded settings. Done means the optimizer can be used through the normal training path and its behavior can be monitored as described.

Written by the indexing model from the issue text.