bigscience-workshop / bigscience-workshop/petals

whether past_key_values can be obtained

Open
#526 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
10.6k
Forks
650
PR merge metrics
No merged PRs in 30d

Description

Hi there.
I'm trying to combine chat.petals.dev with streaming-llm (https://github.com/mit-han-lab/streaming-llm).

However, when streaming-llm needs the output.past_key_values, I found that petals returns [RemotePastKeyValues].
![image](https://github.com/bigscience-workshop/petals/assets/63895563/eeb47f36-839c-40a7-af22-42bc2e402d37)

And the annotation says 'A mock class representing the fact that `past_key_values` do exist but are stored on remote servers.'.

SO, I wonder if there is any way to get the real value of 'past_key_values'.
like this img :
![image](https://github.com/bigscience-workshop/petals/assets/63895563/d6454c68-2089-44c9-a020-26571e5b9c22)

looking for your reply :)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.