microsoft / microsoft/onnxruntime-genai

[DML] Test that destroys a generator, tweaks GeneratorParams and then creates another generator throws a KV_Cache exception

Open
#722 0 comments 0 reactions 0 assignees View on GitHub
ep:DML platform:windows
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

**Describe the bug**
A test that destroys, tweaks GeneratorParams, then creates another instance of a Generator throws a KV_Cache exception.
The error is reproducible only in DML builds

**To Reproduce**

```cpp
TEST(CAPITests, LoraManagement) {
#if defined(USE_CUDA)
const std::string model_folder = MODEL_PATH "hf-internal-testing/tiny-random-llama-lora-fp16";
#elif defined USE_DML
const std::string model_folder = MODEL_PATH "hf-internal-testing/tiny-random-llama-lora-fp16-dml";
#else
const std::string model_folder = MODEL_PATH "hf-internal-testing/tiny-random-llama-lora";
#endif

const std::string adapter_name = "guanaco";

// This should load Lora adapters as configured in the genai_config.json
auto model = OgaModel::Create(model_folder.c_str());

constexpr std::array input_ids_shape{2, 4};
constexpr std::array input_ids{0, 0, 0, 52, 0, 0, 195, 731};
const auto batch_size = input_ids_shape[0];
const auto input_sequence_length = input_ids_shape[1];
constexpr int max_length = 10;

const char* const adapter[] = {adapter_name.c_str()};
auto params = OgaGeneratorParams::Create(*model);
params->SetSearchOption("max_length", max_length);
params->SetInputIDs(input_ids.data(), input_ids.size(), input_sequence_length, batch_size);

// Now we can activate it
// The call validates the adapter names specified.
ASSERT_NO_THROW(params->SetActiveAdapterNames(adapter));

// Try to active a non-existing adapter throws, but preserves the previously set adapters

constexpr const std::array nonexisting_adapter = {"nonexistingadapter"};
ASSERT_THROW(params->SetActiveAdapterNames(nonexisting_adapter), std::runtime_error);

auto generator = OgaGenerator::Create(*model, *params);
ASSERT_NE(nullptr, generator);

generator.reset();

// Reset adapters to base
constexpr std::array base_adapter = {};
params->SetActiveAdapterNames(base_adapter);

generator = OgaGenerator::Create(*model, *params);
ASSERT_NE(nullptr, generator);
}
```

**Expected behavior**
No exception

**Screenshots**
If applicable, add screenshots to help explain your problem.

![image](https://github.com/user-attachments/assets/07cc6b48-5f18-4da7-a04f-5439912187be)
![image](https://github.com/user-attachments/assets/b1e6e0d6-784c-403c-bb32-a5ecc3424997)
![image](https://github.com/user-attachments/assets/0d1dc616-224f-4c70-914b-cfe69795a9b4)

**Desktop (please complete the following information):**
- OS: Windows 11 fully patched
- ORT release 1.18.0 off GH releases page
- GenAI built off the tip
- DirectML 1.14.2
- Direct3d 12.1.614.1

**Additional context**
StaticBuffer instance `bytes_` appears to have a value of 0.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the CAPITests.LoraManagement reproduction in a USE_DML build, then trace the lifecycle of OgaGenerator::Create after generator.reset() and the subsequent SetActiveAdapterNames call. Inspect the StaticBuffer bytes_ state and related KV_Cache handling. Done means the second generator is created without an exception and the regression is covered by the test.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.