InternLM / InternLM/lmdeploy

[Bug] unified_attention split kv for prefill with more workspace coredump

Open
#1,935 3 comments 0 reactions 1 assignee Claimed by @lzhangzz View on GitHub
Dominant language
Python
Stars
8.1k
Forks
748
Avg merge
6d 2h
Merged PRs (30d)
54

Description

### Checklist

- [X] 1. I have searched related issues but cannot get the expected help.
- [X] 2. The bug has not been fixed in the latest version.

### Describe the bug

When using prefix cache, if the input length is very long and the matched prefix is also very long, the efficiency of calculating the prefill in this situation is relatively low. According to the comment 'but only have 1 workspace disable split kv for prefill for now', allocating a separate workspace for prefill still results in a coredump.

### Reproduction

Start the lmdeploy server with any 7B model, output long data, and ensure the prefix easily matches a long sequence. Modify unified_attention to allocate a separate workspace for prefill, then perform a stress test.

### Environment

```Shell
latest version
```

### Error traceback

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.