bigscience-workshop / bigscience-workshop/petals
whether past_key_values can be obtained
- Dominant language
- Python
- Stars
- 10.6k
- Forks
- 650
- PR merge metrics
- No merged PRs in 30d
Description
Hi there.
I'm trying to combine chat.petals.dev with streaming-llm (https://github.com/mit-han-lab/streaming-llm).
However, when streaming-llm needs the output.past_key_values, I found that petals returns [RemotePastKeyValues].

And the annotation says 'A mock class representing the fact that `past_key_values` do exist but are stored on remote servers.'.
SO, I wonder if there is any way to get the real value of 'past_key_values'.
like this img :

looking for your reply :)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.