Subject: Proposal for Implementing a 1-bit Optimizer

Subject: Proposal for Implementing a 1-bit Optimizer

Dear [sd-scripts] Team,

I hope this message finds you well. I am reaching out to propose the addition of a 1-bit optimizer to your project, similar to the implementation discussed on this webpage: https://qiita.com/carrotflakes/items/778f33faca40f32b1aaf. I believe incorporating such a feature could offer significant benefits in terms of reducing both training time and VRAM usage.

During my experiments, I did not utilize BitNet, which the article mentions, to assess the feasibility of the concept independently. My test involved training with a dimension of 16 and an alpha value of 2 (dim16.alpha2). I am pleased to report that the training proceeded to completion without encountering the issue of NaN losses. However, it appears the learning was almost ineffective, as the resulting model showed minimal improvement.

The motivation behind my proposal is to explore whether we can enhance learning efficiency or achieve comparable results with significantly lower computational costs by employing a 1-bit optimization technique. My preliminary experiments suggest that while it is possible to complete the training process, optimizing the approach to achieve effective learning outcomes is necessary.

I am keen to discuss this idea further with your team and explore potential collaborations to refine and implement this concept within your project framework. I believe that with some adjustments and further experimentation, we could unlock substantial benefits for the project, especially in terms of efficiency and resource utilization.

Thank you for considering my proposal. I am looking forward to your feedback and the possibility of contributing to your project.

Best regards,
[kacchan]

Start by reviewing the linked Qiita article and the reporter’s dim16.alpha2 experiment. Identify the project entry points for configuring optimizers and training, which are not named in the issue. Done would require an effective 1-bit optimizer that improves learning while reducing training time or VRAM usage, with results validated against a comparable baseline.

Written by the indexing model from the issue text.