microsoft / microsoft/MInference
[Question]: Top-K Recall Evaluation and Single A100 Inference Details
Open
@iofu728 is already working on this.
Since Jun 19, 2025.
question
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 82
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 1
Description
Describe the issue
Hi, thanks for the great work. I had two quick questions:
-
In the paper, it’s mentioned that retaining top-4096 attention columns achieves 96.4% recall under a 128K context. Could you share more details on how this was implemented? Given that the full attention matrix at this scale would not fit into GPU memory, how did you handle the memory issue during evaluation?
-
In Appendix C.3 (Single A100 Implementation Details), would it be possible to release the test code used for the 1M-token inference on a single A100?
Thanks again!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.