[Feature]: miniCOIL: no way to control input sequence length (and VRAM usage) natively
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3.2k
- Forks
- 248
- Avg merge
- 4d 8h
- Merged PRs (30d)
- 4
Description
What feature would you like to request?
As I understand it, the interface of the miniCOIL model does not allow users to control the length of tokens passed to the base transformer model. As miniCOIL uses an 8k context model as a backbone, users will be suddenly running out of GPU memory when processing large texts.
Adding a max_sequence_length or similar parameter would be nice.
An easy option to avoid OOMs is to truncate the input string, but it is wrong on many levels.
Is there any additional information you would like to provide?
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the miniCOIL model interface and the code that passes tokens to its 8k-context base transformer. Determine how a max_sequence_length-style option should bound transformer input without simply truncating the input string; done means the option controls input length and avoids unexpected VRAM use.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100