bottlerocket-os / bottlerocket-os/bottlerocket
Feature Request: Include Lustre client fix for statahead umount deadlock (LU-14883 / LU-18602) — commit f4bf36d
- Dominant language
- Rust
- Stars
- 9.7k
- Forks
- 586
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 11
Description
## Summary
We are running Bottlerocket v1.57.0 on EKS with an Amazon FSx for Lustre filesystem mounted on our worker nodes. We are hitting two confirmed upstream Lustre bugs that cause **node hangs during unmount** due to a statahead deadlock. Both bugs are fixed by a single upstream commit in `lustre/lustre-release` that we would like to see reflected in the Lustre client shipped with or supported by Bottlerocket.
---
## Affected Versions
| Component | Version |
|-----------|---------|
| Bottlerocket | v1.57.0 |
| bottlerocket-kernel-kit | v5.2.0 |
| Kernels affected | 6.1.x, 6.12.x |
| Lustre | FSx for Lustre (Lustre client) |
| EKS | Yes |
---
## The Bugs
### LU-14883 — umount blocked due to remaining statahead threads
**JIRA:** https://jira.whamcloud.com/browse/LU-14883
**Priority:** Major
**Status:** Open
During unmount, multiple `ll_sa` (statahead) worker threads remain blocked inside `ll_statahead_thread()` calling `schedule()`, even though no active jobs are running. The `ll_sa_running` counter stays non-zero (e.g. 3), which causes the umount process to block indefinitely in `ll_kill_super()`.
Observed stack trace pattern:
```
ll_statahead_thread [lustre]
kthread
```
The `ll_umounting` flag is set but the statahead threads never observe it and exit cleanly, resulting in a permanent deadlock on the node.
---
### LU-18602 — client deadlock at umount (statahead async enqueue + inode release)
**JIRA:** https://jira.whamcloud.com/browse/LU-18602
**Priority:** Minor
**Status:** Resolved (fixed in Lustre 2.17.0)
During unmount, the kernel attempts to release all inodes. A Lustre object representing an inode can be pinned by an asynchronous enqueue RPC initiated by statahead. If the OSC import is not active (e.g. an OST is offline, or the mount is being torn down), a deadlock occurs because:
> "nobody invalidates the import and `ll_clear_inode()` is waiting till the object is freed."
Call chain:
```
unmount → evict_inodes → ll_delete_inode → ll_clear_inode
→ lov_object_delete → lov_delete_composite → [LBUG assertion]
```
This results in a node that cannot complete its umount and must be forcibly terminated.
---
## The Fix
Both issues are addressed by the following upstream commit in [`lustre/lustre-release`](https://github.com/lustre/lustre-release):
**Commit:** [`f4bf36d8f4957b62e4edab58b971829dccedd0c0`](https://github.com/lustre/lustre-release/commit/f4bf36d8f4957b62e4edab58b971829dccedd0c0)
**Author:** Qian Yingjin \
**Date:** 2023-07-06
**Reviewed on:** https://review.whamcloud.com/c/fs/lustre-release/+/51592
**Reviewers:** Lai Siyao, Andreas Dilger, Oleg Drokin
**Commit title:** `LU-14361 statahead: add tunable for fname pattern detection`
This commit contains the following fixes directly relevant to LU-14883 and LU-18602:
1. **Fixes panic in `ll_sax_put` → `ll_statahead_thread`** — caused by `@lli_sax` being set to NULL by the wrong statahead context (sax) put.
2. **Fixes possible deadlock between hardlink and batch statahead operations.**
3. **Forces umount to wait for all in-use inodes held by statahead to be released before proceeding**, preventing the following crash:
```
BUG: Dentry 0000000033ca4f3e{i=280001b840002c7,n=l4} still in use (1) [unmount of lustre lustre]
RIP: 0010:umount_check.cold.52+0x2f/0x3b
d_walk+0xe7/0x290
do_one_tree+0x20/0x40
shrink_dcache_for_umount+0x28/0x90
generic_shutdown_super+0x1a/0x110
kill_anon_super+0x14/0x30
deactivate_locked_super+0x34/0x70
cleanup_mnt+0x3b/0x70
```
4. **Fixes statahead thread not stopping**, which is a direct root cause of the LU-14883 umount hang.
---
## Impact on Bottlerocket / EKS Users
- Bottlerocket nodes mounting FSx for Lustre **cannot cleanly unmount** when a pod or node is terminated, because statahead threads are left dangling.
- This causes EKS node drain operations to hang, Kubernetes evictions to stall, and in worst cases nodes must be force-terminated (causing possible data consistency concerns).
- Statahead is enabled by default in the Lustre client and is used heavily in HPC/ML workloads.
---
## Request
We would appreciate one of the following:
1. **Cherry-pick** commit `f4bf36d8f4957b62e4edab58b971829dccedd0c0` into the Lustre client kernel module version shipped with or supported by Bottlerocket.
2. **Bump** the Lustre client to a version that includes this fix (Lustre ≥ 2.17.0 includes LU-18602; the statahead umount fix from LU-14361/f4bf36d should be present in the corresponding Lustre version built against your kernel).
3. If you are already tracking a newer Lustre client that includes this fix, **please confirm** so we know which Bottlerocket version resolves the issue.
---
## References
- Upstream commit: https://github.com/lustre/lustre-release/commit/f4bf36d8f4957b62e4edab58b971829dccedd0c0
- Whamcloud Gerrit review: https://review.whamcloud.com/c/fs/lustre-release/+/51592
- LU-14883 (JIRA): https://jira.whamcloud.com/browse/LU-14883
- LU-18602 (JIRA): https://jira.whamcloud.com/browse/LU-18602
- Bottlerocket kernel kit: https://github.com/bottlerocket-os/bottlerocket-kernel-kit
Contributor guide
Assessment
This issue has not been assessed yet.