bigscience-workshop / bigscience-workshop/petals

Allow filtering by max sequence length

Open
#394 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
10.6k
Forks
650
PR merge metrics
No merged PRs in 30d

Description

Problem: if some (but not all) servers support longer sequence length, inferencing with that sequence length would be very inefficient because the client will constantly bump into short-length servers.

Suggested solution: if we ask servers to report max sequence length to the DHT, a client will be able to filter by sequence length as they read DHT entries.

Contributor guide

No contributing guide indexed for this repository

Research direction

No files or tests are named in the issue. Start by tracing how servers publish capabilities to DHT entries and how clients select entries for inference; done means clients can filter out servers whose reported maximum sequence length is too short.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.