kvcache-ai / kvcache-ai/ktransformers
unit tests
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 1.6k
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 27
Description
I've noticed commit [8dc1ab9](https://github.com/kvcache-ai/ktransformers/commit/8dc1ab9) (Apr 14th) is WAY more reliable on my hardware with `DeepSeek-V3-0324:671b-q4_k_m` than current [64ec0ec](https://github.com/kvcache-ai/ktransformers/commit/64ec0ec148b472850bb9145161da43947508d9f7) (June 11th). I can get 50k to 80k long context from [8dc1ab9](https://github.com/kvcache-ai/ktransformers/commit/8dc1ab9). However, I'm lucky to get 24k to 29k long context from [64ec0ec](https://github.com/kvcache-ai/ktransformers/commit/64ec0ec148b472850bb9145161da43947508d9f7).
My hardware is dual EPYC 9355 CPUs with 768gb RAM and a 24gb 3090.
I've also noticed that I can easily write code that runs, but introduces a lot of instability and causes models to crash. For instance, I wrote code that backported R1-0528 support all the way to the March 15th commit `bda9cf1` - it ran, but it was useless because it went insane after just a turn or two.
How can we/I write unit tests that will help us verify that we aren't introducing problems when we make changes?
Contributor guide
Assessment
This issue has not been assessed yet.