kvcache-ai / kvcache-ai/ktransformers

unit tests

Open
#1,381 5 comments 0 reactions 1 assignee Claimed by @Azure-Tang View on GitHub
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

I've noticed commit [8dc1ab9](https://github.com/kvcache-ai/ktransformers/commit/8dc1ab9) (Apr 14th) is WAY more reliable on my hardware with `DeepSeek-V3-0324:671b-q4_k_m` than current [64ec0ec](https://github.com/kvcache-ai/ktransformers/commit/64ec0ec148b472850bb9145161da43947508d9f7) (June 11th). I can get 50k to 80k long context from [8dc1ab9](https://github.com/kvcache-ai/ktransformers/commit/8dc1ab9). However, I'm lucky to get 24k to 29k long context from [64ec0ec](https://github.com/kvcache-ai/ktransformers/commit/64ec0ec148b472850bb9145161da43947508d9f7).

My hardware is dual EPYC 9355 CPUs with 768gb RAM and a 24gb 3090.

I've also noticed that I can easily write code that runs, but introduces a lot of instability and causes models to crash. For instance, I wrote code that backported R1-0528 support all the way to the March 15th commit `bda9cf1` - it ran, but it was useless because it went insane after just a turn or two.

How can we/I write unit tests that will help us verify that we aren't introducing problems when we make changes?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.