huggingface / huggingface/peft

[RFC, I can do a PR] Make `lora_alpha` a float

Open
#3,615 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
21.7k
Forks
2.5k
Avg merge
4d 12h
Merged PRs (30d)
59

Description

### Feature request

The code generally types `lora_alpha` as `int`. As far as I can tell, there's nothing fundamentally non-continuous or integer about `lora_alpha`; on the other hand, for hyperparameter optimization (Bayesian methods, hyperopt, Ax, etc.), needlessly discrete cases are often harder to handle.

On the other hand, this may be mostly a typing-only issue, or typing + documentation issue, and it's not clear if anything should be done; see discussion below.

### Your contribution

I'm willing to send a PR, but first I think it should be established whether, and in what form, it's desirable.

Now, I'm mindful of the admonishment in CONTRIBUTING.md to not submit typing-only changes because they cause more work than they save; and the code seems to handle, as far as I can tell `float` lora_alpha just fine.

From a downstream of a downsteam perspective, that does mean something of an uphill to convince another project to pass a float to peft when its interface asks for an int. In a sense, typing has two roles here: Maintaining internal typing consistency and documenting the external interface. The latter could, of course, also be taken care of by other forms of documentation or comment indicating upstream blessing (such as "sounds good but probably not worth the typing churn") that would hopefully be more convincing than "I tried it and it seems to not immediately break".

So, I can offer to send a PR changing the types, but would that be desirable?

Contributor guide

Open the contributing guide

Research direction

The issue does not name any files, tests, or entry points. First inspect where lora_alpha is typed and documented, then review the linked discussion to establish whether float support is desired; done requires an explicit maintainer decision on the scope before any PR is started.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.