microsoft / microsoft/onnxruntime-genai
[DML] Test that destroys a generator, tweaks GeneratorParams and then creates another generator throws a KV_Cache exception
- Dominant language
- C++
- Stars
- 1.1k
- Forks
- 354
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 85
Description
**Describe the bug**
A test that destroys, tweaks GeneratorParams, then creates another instance of a Generator throws a KV_Cache exception.
The error is reproducible only in DML builds
**To Reproduce**
```cpp
TEST(CAPITests, LoraManagement) {
#if defined(USE_CUDA)
const std::string model_folder = MODEL_PATH "hf-internal-testing/tiny-random-llama-lora-fp16";
#elif defined USE_DML
const std::string model_folder = MODEL_PATH "hf-internal-testing/tiny-random-llama-lora-fp16-dml";
#else
const std::string model_folder = MODEL_PATH "hf-internal-testing/tiny-random-llama-lora";
#endif
const std::string adapter_name = "guanaco";
// This should load Lora adapters as configured in the genai_config.json
auto model = OgaModel::Create(model_folder.c_str());
constexpr std::array input_ids_shape{2, 4};
constexpr std::array input_ids{0, 0, 0, 52, 0, 0, 195, 731};
const auto batch_size = input_ids_shape[0];
const auto input_sequence_length = input_ids_shape[1];
constexpr int max_length = 10;
const char* const adapter[] = {adapter_name.c_str()};
auto params = OgaGeneratorParams::Create(*model);
params->SetSearchOption("max_length", max_length);
params->SetInputIDs(input_ids.data(), input_ids.size(), input_sequence_length, batch_size);
// Now we can activate it
// The call validates the adapter names specified.
ASSERT_NO_THROW(params->SetActiveAdapterNames(adapter));
// Try to active a non-existing adapter throws, but preserves the previously set adapters
constexpr const std::array nonexisting_adapter = {"nonexistingadapter"};
ASSERT_THROW(params->SetActiveAdapterNames(nonexisting_adapter), std::runtime_error);
auto generator = OgaGenerator::Create(*model, *params);
ASSERT_NE(nullptr, generator);
generator.reset();
// Reset adapters to base
constexpr std::array base_adapter = {};
params->SetActiveAdapterNames(base_adapter);
generator = OgaGenerator::Create(*model, *params);
ASSERT_NE(nullptr, generator);
}
```
**Expected behavior**
No exception
**Screenshots**
If applicable, add screenshots to help explain your problem.



**Desktop (please complete the following information):**
- OS: Windows 11 fully patched
- ORT release 1.18.0 off GH releases page
- GenAI built off the tip
- DirectML 1.14.2
- Direct3d 12.1.614.1
**Additional context**
StaticBuffer instance `bytes_` appears to have a value of 0.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the CAPITests.LoraManagement reproduction in a USE_DML build, then trace the lifecycle of OgaGenerator::Create after generator.reset() and the subsequent SetActiveAdapterNames call. Inspect the StaticBuffer bytes_ state and related KV_Cache handling. Done means the second generator is created without an exception and the regression is covered by the test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100