iree-org / iree-org/fusilli

[SDPA][hipDNN] Dropout not supported

Open
#277 6 comments 0 reactions 1 assignee Assigned to @rsuderman View on GitHub
Dominant language
C++
Stars
25
Forks
16
PR merge metrics
No merged PRs in 30d

Description

SDPA graphs with `dropout_probability > 0` are rejected by the fusilli plugin.

We're integrating hipDNN SDPA into PyTorch. To reach feature parity with cuDNN and pass the upstream `test_cudnn_attention_*` tests, the fusilli plugin would need to support this. [`test_cudnn_attention_seqlen1_dropout_heuristic`](https://github.com/pytorch/pytorch/blob/6035abb8d168a7e2c9524a86efa2b976d178ccca/test/test_transformers.py#L3105-L3111) calls `scaled_dot_product_attention` with `dropout_p=0.5` .

**Repro:**

Graph serialized from `test_cudnn_attention_seqlen1_dropout_heuristic` via `mha_graph->toJson()`. Apply to `dnn-providers/fusilli-provider/test/integration/sdpa/simple_sdpa.cpp`:

```diff
--- a/dnn-providers/fusilli-provider/test/integration/sdpa/simple_sdpa.cpp
+++ b/dnn-providers/fusilli-provider/test/integration/sdpa/simple_sdpa.cpp
@@ -18,0 +19,2 @@
+#include
+
@@ -137,0 +140,70 @@ TEST(SdpaIntegrationTest, SimpleSdpa) {
+// Graph from PyTorch test_cudnn_attention_seqlen1_dropout_heuristic (f16, B=2 H=8 S=1 D=128).
+// Fails at create_execution_plans because dropout_probability > 0 is not supported.
+TEST(SdpaIntegrationTest, Dropout) {
+ SKIP_IF_NO_DEVICES();
+ ASSERT_EQ(hipInit(0), hipSuccess);
+
+ auto pluginPath = std::filesystem::canonical(getCurrentExecutableDirectory() /
+ FUSILLI_PLUGIN_PATH);
+ const std::array paths = {pluginPath.c_str()};
+ ASSERT_EQ(hipdnnSetEnginePluginPaths_ext(paths.size(), paths.data(),
+ HIPDNN_PLUGIN_LOADING_ABSOLUTE),
+ HIPDNN_STATUS_SUCCESS);
+ hipdnnHandle_t handle;
+ ASSERT_EQ(hipdnnCreate(&handle), HIPDNN_STATUS_SUCCESS);
+
+ // clang-format off
+ auto graph = std::make_shared();
+ auto result = graph->deserialize(nlohmann::json::parse(R"({
+ "compute_data_type": "float",
+ "intermediate_data_type": "float",
+ "io_data_type": "half",
+ "name": "",
+ "tensors": [
+ { "uid": 0, "name": "Q", "data_type": "unset", "dims": [2, 8, 1, 128], "strides": [1024, 128, 128, 1], "virtual": false },
+ { "uid": 1, "name": "K", "data_type": "unset", "dims": [2, 8, 1, 128], "strides": [1024, 128, 128, 1], "virtual": false },
+ { "uid": 2, "name": "V", "data_type": "unset", "dims": [2, 8, 1, 128], "strides": [1024, 128, 128, 1], "virtual": false },
+ { "uid": 3, "name": "CUDNN_SDPA::O", "data_type": "unset", "dims": [2, 8, 1, 128], "strides": [1024, 128, 128, 1], "virtual": false },
+ { "uid": 5, "name": "Attn_scale", "data_type": "float", "dims": [1], "strides": [1], "value": 0.0883883461356163, "value_type": "Float32Value", "virtual": false },
+ { "uid": 6, "name": "Seed", "data_type": "int32", "dims": [1, 1, 1, 1], "strides": [1, 1, 1, 1], "virtual": false },
+ { "uid": 7, "name": "Offset", "data_type": "int32", "dims": [1, 1, 1, 1], "strides": [1, 1, 1, 1], "virtual": false },
+ { "uid": 8, "name": "CUDNN_SDPA::STATS","data_type": "float", "dims": [2, 8, 1, 1], "strides": [8, 1, 1, 1], "virtual": false }
+ ],
+ "nodes": [{
+ "type": "SdpaAttributes", "name": "CUDNN_SDPA", "compute_data_type": "unset",
+ "attributes": {
+ "dropout_probability": 0.5,
+ "generate_stats": true, "causal_mask": false,
+ "alibi_mask": false, "attn_scale_value": null, "causal_mask_bottom_right": false,
+ "diagonal_alignment": "TOP_LEFT", "implementation": "AUTO",
+ "left_bound": null, "max_seq_len_kv": null, "mma_core_mode": "unset",
+ "padding_mask": false, "right_bound": null
+ },
+ "inputs": {
+ "q_tensor_uid": 0, "k_tensor_uid": 1, "v_tensor_uid": 2, "scale_tensor_uid": 5,
+ "seed_tensor_uid": 6, "offset_tensor_uid": 7,
+ "attn_mask_tensor_uid": null, "block_mask_tensor_uid": null,
+ "descale_k_tensor_uid": null, "descale_q_tensor_uid": null,
+ "descale_s_tensor_uid": null, "descale_v_tensor_uid": null,
+ "dropout_mask_tensor_uid": null, "dropout_scale_tensor_uid": null,
+ "page_table_k_tensor_uid": null, "page_table_v_tensor_uid": null,
+ "scale_o_tensor_uid": null, "scale_s_tensor_uid": null,
+ "seq_len_kv_tensor_uid": null, "seq_len_q_tensor_uid": null,
+ "sink_token_tensor_uid": null
+ },
+ "outputs": {
+ "o_tensor_uid": 3, "stats_tensor_uid": 8,
+ "amax_o_tensor_uid": null, "amax_s_tensor_uid": null,
+ "max_tensor_uid": null, "rng_dump_tensor_uid": null,
+ "sum_exp_tensor_uid": null
+ }
+ }]
+ })"));
+ // clang-format on
+ ASSERT_EQ(result.code, error_code_t::OK);
+ ASSERT_EQ(graph->validate().code, error_code_t::OK);
+ ASSERT_EQ(graph->build_operation_graph(handle).code, error_code_t::OK);
+ // Fails: "No engine configurations available for the graph."
+ ASSERT_EQ(graph->create_execution_plans({HeurMode_t::FALLBACK}).code, error_code_t::OK);
+}
+
```

**Error output:**

`create_execution_plans` returns `HIPDNN_BACKEND_ERROR`:
```
No engine configurations available for the graph.
```

With `HIPDNN_LOG_LEVEL=info`:
```
[FUSILLI_ENGINE] Graph not supported: SDPA with dropout not supported.
```

**Environment:**

- fusilli-provider from `rocm-libraries` at `8511c3d320`
- fusilli at `6165d53`
- ROCm 7.1.0, MI300X (gfx942)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.