microsoft / microsoft/MInference
[Question]: Am I using minference correctly?
@iofu728 is already working on this.
Since Nov 4, 2024.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 82
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 1
Description
Describe the issue
Hi,
I followed the instructions for running minference on HF and ran an example where I give it the full text of Dante's inferno - in Italian. As well as the book of the harry potter series, and then ask it a few questions. I'm testing this on the llama 3p1 8b instruct model, but with the config modified so the sequence length is 262k.
https://gist.github.com/YLGH/2b70d6ed10a6b5ea97404cb2668e24f3
The output when using minference attention seems to be completely off. It doesn't acknowledge the existence of Dante's inferno at all, and says that I gave it books 1 and 2 of harry potter.
I also ran the same prompt through full dense attention, and it's able to distinguish the two.
Am I using the HF example correctly?
Thanks!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.