huggingface / huggingface/peft
[RFC, I can do a PR] Make `lora_alpha` a float
- Dominant language
- Python
- Stars
- 21.7k
- Forks
- 2.5k
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 59
Description
### Feature request
The code generally types `lora_alpha` as `int`. As far as I can tell, there's nothing fundamentally non-continuous or integer about `lora_alpha`; on the other hand, for hyperparameter optimization (Bayesian methods, hyperopt, Ax, etc.), needlessly discrete cases are often harder to handle.
On the other hand, this may be mostly a typing-only issue, or typing + documentation issue, and it's not clear if anything should be done; see discussion below.
### Your contribution
I'm willing to send a PR, but first I think it should be established whether, and in what form, it's desirable.
Now, I'm mindful of the admonishment in CONTRIBUTING.md to not submit typing-only changes because they cause more work than they save; and the code seems to handle, as far as I can tell `float` lora_alpha just fine.
From a downstream of a downsteam perspective, that does mean something of an uphill to convince another project to pass a float to peft when its interface asks for an int. In a sense, typing has two roles here: Maintaining internal typing consistency and documenting the external interface. The latter could, of course, also be taken care of by other forms of documentation or comment indicating upstream blessing (such as "sounds good but probably not worth the typing churn") that would hopefully be more convincing than "I tried it and it seems to not immediately break".
So, I can offer to send a PR changing the types, but would that be desirable?
Contributor guide
Research direction
The issue does not name any files, tests, or entry points. First inspect where lora_alpha is typed and documented, then review the linked discussion to establish whether float support is desired; done requires an explicit maintainer decision on the scope before any PR is started.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100