bigscience-workshop / bigscience-workshop/petals
Allow filtering by max sequence length
- Dominant language
- Python
- Stars
- 10.6k
- Forks
- 650
- PR merge metrics
- No merged PRs in 30d
Description
Problem: if some (but not all) servers support longer sequence length, inferencing with that sequence length would be very inefficient because the client will constantly bump into short-length servers.
Suggested solution: if we ask servers to report max sequence length to the DHT, a client will be able to filter by sequence length as they read DHT entries.
Contributor guide
No contributing guide indexed for this repository
Research direction
No files or tests are named in the issue. Start by tracing how servers publish capabilities to DHT entries and how clients select entries for inference; done means clients can filter out servers whose reported maximum sequence length is too short.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100