awslabs / awslabs/keys_values

Identify and fix issues with `bnb-quantized*` quantization

Open
#11 0 comments 0 reactions 0 assignees View on GitHub
enhancement good first issue
Dominant language
Python
Stars
18
Forks
8
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**

There are issues with `bnb-quantized*` quantization. For example, in the test `kvcache/test_quantization.py -k test_quantization_error`,
we skip `blocks_over_heads = True, name = 'dense-bnb-quantized*`.

**To reproduce**

Remove the skip in the test and run it (on GPU).

**Expected behavior**

`bnb-quantized*`should either work or be removed.

Contributor guide

Open the contributing guide

Research direction

Start with kvcache/test_quantization.py and remove the skip for blocks_over_heads = True, name = 'dense-bnb-quantized*'. Run test_quantization_error on a GPU to observe the failure and trace the affected quantization path. Done means the test passes for the quantized variant, or the variant is removed if it cannot work.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.