Repeated kernel Oops in __wake_up_common from ZFS QAT compression callback
- Dominant language
- C
- Stars
- 169
- Forks
- 59
- PR merge metrics
- No merged PRs in 30d
Description
### System information
| Item | Value |
|---|---|
| Distribution / kernel | RHEL/CentOS 7-derived, `3.10.0-693.el7.x86_64` |
| Architecture | x86_64 |
| ZFS / SPL | `0.6.5.11-1` |
| QAT devices | 3 × Intel C62x devices (`qat_dev0`–`qat_dev2`) |
| QAT modules in crash stack | `qat_api`, `intel_qat`, `qat_c62x` |
| Workload | KVM/ZVOL storage workload; multiple ZFS `zvol` and `metaslab_group_` threads have been observed in D state |
| Kernel command line | `intel_iommu=on iommu=pt ... numa=off` |
| QAT/IOMMU boot message | Each C62x device reports `Cannot use PF with IOMMU enabled` |
### Problem
We have two verified kdump crashes with the same call chain. The crashes occurred at:
- 2026-08-19 01:12:55, after approximately 76 days 19 hours of uptime.
- 2026-08-26 01:59:17, after approximately 7 days of uptime.
Both crashes are a kernel page fault in `__wake_up_common()` while `complete()` is called from the ZFS QAT data-compression callback.
The faulting address differs between crashes and is a non-canonical/low address. In each case `RDI` is a valid kernel address for the wait queue head, while `RDX` and `CR2` contain the invalid address. This appears consistent with corruption or premature reuse of the `completion.wait.task_list` list, but we cannot yet establish whether the source is a ZFS lifetime race, QAT callback timing, a duplicate completion, or unrelated memory corruption.
### Crash 1: 2026-08-19 01:12:55
```text
[6635725.272058] BUG: unable to handle kernel paging request at 000000005d38807c
[6635725.272657] IP: [] __wake_up_common+0x2b/0x90
[6635725.273675] Oops: 0000 [#1] SMP
[6635725.281299] CPU: 0 PID: 10599 Comm: z_rd_int_1 Tainted: P OE ------------ 3.10.0-693.el7.x86_64 #1
[6635725.284906] RDX: 000000005d38807c RSI: 0000000000000003 RDI: ffff885d4c16fc20
[6635725.288632] CR2: 000000005d38807c
[6635725.294570] [] complete+0x3c/0x50
[6635725.295272] [] qat_dc_callback+0x13/0x20 [zfs]
[6635725.295967] [] dcCompression_ProcessCallback+0x1f8/0x770 [qat_api]
[6635725.298066] [] adf_handle_response+0x4b/0xc0 [intel_qat]
[6635725.298713] [] adf_response_handler+0x90/0xc0 [intel_qat]
```
### Crash 2: 2026-08-26 01:59:17
```text
[605780.815602] BUG: unable to handle kernel paging request at 000000002edd7959
[605780.815637] IP: [] __wake_up_common+0x2b/0x90
[605780.815677] Oops: 0000 [#1] SMP
[605780.817150] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: P OE ------------ 3.10.0-693.el7.x86_64 #1
[605780.822008] RDX: 000000002edd7959 RSI: 0000000000000003 RDI: ffff885d5ba83c20
[605780.826910] CR2: 000000002edd7959
[605780.833205] [] complete+0x3c/0x50
[605780.834009] [] qat_dc_callback+0x13/0x20 [zfs]
[605780.834783] [] dcCompression_ProcessCallback+0x1f8/0x770 [qat_api]
[605780.837085] [] adf_handle_response+0x4b/0xc0 [intel_qat]
[605780.837842] [] adf_response_handler+0x90/0xc0 [intel_qat]
```
### Related symptoms
Before both crashes, ZFS reports hung tasks for more than 120 seconds:
- Crash 1: `z_wr_iss`, two `metaslab_group_` threads, `txg_sync`, and six `zvol` threads.
- Crash 2: two `metaslab_group_` threads and eight `zvol` threads.
After the 2026-08-26 reboot, two `metaslab_group_` threads again blocked for more than 120 seconds after about nine hours. We found no MCE, ECC, PCIe AER, block I/O error, OOM, or watchdog-panic evidence in the supplied logs.
### Questions
1. Is a lifetime race, delayed callback, or duplicate completion involving `qat_dc_callback()` and its `completion` object known in the ZFS 0.6.x QAT integration?
2. Is ZFS 0.6.5.11 QAT support compatible with Intel C62x PF devices while `intel_iommu=on iommu=pt` is enabled? The QAT driver logs `Cannot use PF with IOMMU enabled` for all three devices.
3. Is setting `zfs_qat_compress_disable=1` the supported immediate mitigation for this crash path, and does it prevent only new submissions or also safely drain/handle outstanding compression requests?
4. Which exact versions of ZFS/SPL, QAT driver/firmware, and kernel are supported together for C62x hardware on an EL7-era 3.10 kernel?
5. Which additional artifacts or debug instrumentation would maintainers need to identify whether the corrupted wait queue was freed, reused, overwritten, or completed twice?
### Artifacts available on request
- Original vmcores for both dates.
- Matching `vmcore-dmesg` files.
- QAT service/device configuration and module metadata.
- ZFS pool status and QAT counters collected after approval.
The vmcores may contain sensitive workload metadata and will not be uploaded to a public issue without review.
```
## Official-site search results
### Related, but not the same
1. [OpenZFS issue #7081](https://github.com/openzfs/zfs/issues/7081) documents QAT compression integration on an EL7-era 3.10 kernel and checks `/proc/spl/kstat/zfs/qat` counters. It is an activation/usage issue, **not** a callback Oops.
2. [OpenZFS issue #8600](https://github.com/openzfs/zfs/issues/8600) reports a page fault in a ZFS compression function under heavy I/O on ZFS 0.7.x. It is relevant as an older compression-path crash, but it does **not** include QAT or this callback chain.
3. [Intel QATlib's official known-issues page](https://github.com/intel/qatlib) lists current data-compression issues. No public, indexed issue matching `dcCompression_ProcessCallback → qat_dc_callback → complete → __wake_up_common` was found during this search.
### Search conclusion
No publicly indexed OpenZFS or Intel QAT report with the exact stack was found. This does not prove no historical/internal vendor issue exists. The target stack is based on legacy ZFS 0.6.5.11 and an EL7 3.10 kernel, so maintainers will likely require the exact vendor module build, QAT driver/firmware version, and original vmcore before they can compare it with known defects.
[vmcore-dmesg_202608190112.txt](https://github.com/user-attachments/files/31630519/vmcore-dmesg_202608190112.txt)
[vmcore-dmesg_202608260159.txt](https://github.com/user-attachments/files/31630520/vmcore-dmesg_202608260159.txt)
Contributor guide
Research direction
Start with the two supplied vmcore-dmesg files and the qat_dc_callback → complete → __wake_up_common stack, then compare the ZFS 0.6.5.11 QAT integration with the C62x, IOMMU, kernel, driver, and firmware details. Request the original vmcores and exact module and device versions. Done means identifying whether the completion was freed, reused, overwritten, or duplicated, and documenting a supported mitigation and version combination.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, linux
- Domain
- operating-systems, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100