[BUG] Cluster stops accepting writes when one hot node hangs

Describe the bug

Description:
We are running an OpenSearch cluster with the following setup:

• OS: RHEL 9 on ESX cluster
• Security: Secure Boot enabled
• Security agents installed: TrendMicro & CrowdStrike Falcon
• Cluster size: 32 hot nodes + several warm nodes + several coordinating nodes + several search nodes + several content nodes + 3 master nodes = total of 226 nodes
• Monitoring: Telegraf agent on each node

What happened:
Earlier today, one of our hot nodes became completely unresponsive due to a kernel lockdown event.
We suspect this may be related to TrendMicro, CrowdStrike, or both, but we are still waiting for confirmation from their support teams.

While this single hot node was frozen, the entire cluster stopped accepting write requests with write timeouts.
This was unexpected, as we assumed the remaining 31 healthy hot nodes could still handle writes.

What we observed:

• The problematic node still appeared as part of the cluster according to OpenSearch.
• However, it was not sending any telemetry to Telegraf.
• There was no relevant error message in the OpenSearch logs of the affected node.
• Other nodes also showed no obvious errors in logs.
• We cannot ssh to the problematic node.
• We disconnected the node from the network. Only then the cluster started to accept write requests and reallocating the shards.
• After isolating the node for several hours, we reconnected its network and it successfully joined to the cluster without any problem and started operating normally.

Our question:

• Why would the entire cluster reject write requests when only one hot node was hung?
• Does OpenSearch block writes if an unresponsive hot node still appears connected?
• Is there a recommended way to configure node timeouts or fault detection so that a single stuck node doesn’t affect the whole cluster?

Our OpenSearch version is 2.19 and JVM version is 11.

Related component

Indexing

To Reproduce

1. Install TrendMicro agent.
2. Install falcon-sensor-bpf of Crowstrike.
3. Enable secure boot.
4. If you are lucky, sometimes you get a kernel lockdown.

Expected behavior

If a single hot node becomes unresponsive (e.g., due to kernel hang), the cluster should detect the failure within the configured fault detection timeout and automatically reroute shard assignments so that write requests can continue to be processed by the remaining healthy hot nodes.

Additional Details

Plugins
No extra plugins are installed.

Screenshots
Cannot do that due to the level of confidentiality.

Host/Environment (please complete the following information):

• OS: RHEL 9
• Version 2.19

Additional context
All described in the previous section.

No source file, test, or specific entry point is identified. Start by reviewing OpenSearch 2.19 fault detection, shard allocation, and indexing behavior, then determine whether the reported node hang can be reproduced and what logs or cluster state explain the write timeouts.