bigscience-workshop / bigscience-workshop/petals

Miscellaneous server-side improvements

Open
#68 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
10.6k
Forks
650
PR merge metrics
No merged PRs in 30d

Description

* task_pool
* [x] remove interate_minibatches, once https://github.com/learning-at-home/hivemind/pull/506 is merged
* [ ] batch sequences of similar length

* runtime
* [x] do not log "0 parameters" on init (misleading)
* [x] consider removing Runtime, once https://github.com/learning-at-home/hivemind/pull/505 is merged

* handler
* [x] verify what happens if server have mixed torch_dtype and/or compression
* [x] actually follow forward/backward/inference schema instead of hard-coding
* [x] extract the code for adding prompts into a separate file
* [x] consider merging the code from hivemind's ConnectionHandler instead of inheriting
* [x] add a test that covers _rpc_inference with prompts
* [x] add a test that covers _rpc_inference with hypo-ids

* MemoryCache
* [x] when running inference over multiple layers on the same server, avoid passing layer activations between cpu<->gpu by
storing them in MemoryCache
- before implementing this, gotta check if this will bring any performance benefit
* [ ] LRU-offload stale cache from gpu to ram

* point system
* [x] make sure points are integers everywhere
* [ ] implement a nonzero prioritizer :)
* [x] move client-side spending polity to sequence_manager

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.