microsoft / microsoft/Windows-Containers
Requesting elaboration for HCN error 2151350293 (0x803b0015)
@grcusanz is already working on this.
Since Apr 15, 2024.
- Dominant language
- PowerShell
- Stars
- 551
- Forks
- 76
- PR merge metrics
- No merged PRs in 30d
Description
Summary
I am requesting more details about the HCN error code 2151350293 (0x803b0015). In a scenario I am facing, this error is returned during the HcnDeleteNamespace operation when invoked by containerd via hcsshim.
To help me troubleshoot an ongoing issue and perhaps also benefit the broader community,
- in the short term,
- Could the team please provide a more detailed explanation of the error code 2151350293 including information on what scenarios are likely synonymous with it? We would greatly appreciate understanding the scenarios that might trigger this error, as having more detail about this error will be helpful in diagnosing and resolving issues more efficiently.
- for future improvements,
- Would it be possible to enhance the error messages to be more specific and add new specific error codes if necessary? Less generic messages could significantly help provide clearer context.
Further detail
When rapidly scaling containers (up and down) on a Kubernetes Windows node running containerd, I am sporadically encountering errors while trying to terminate Kubernetes pods. The pods get stuck in terminating status while the containerd tries to remove the network namespace using the HcnDeleteNamespace operation. The error message in the logs is as follows:
hcnDeleteNamespace failed in Win32: The specified request is unsupported. (0x803b0015)
{
"Success": false,
"Error": "The specified request is unsupported.",
"ErrorCode": 2151350293
}
More context
- The error is more prevalent during high CPU usage caused by simultaneous container creation and deletion, although it has also occurred under normal conditions without high resource utilization.
- I observed this error in environments using bot Windows Server 2019 and 2022. I also observed this error in multiple kubernetes and containerd versions but my recent focus is on testing on using environment described below.
- To reproduce the issue, it occurs sporadically when I rapidly terminate and create containers to the maximum capacity of the node. It is notable to mention that this error occurs more prevalently when using an Amazon EKS feature called prefix delegation whereby the node is assigned IP prefixes which in practical terms means you can run as many pods as the node's resources can handle (that is, IP assignments are not a limitation to pod scaling)
Current theories being explored for which more error verbosity can help rule out
- Potential issues in the HCN networking stack
- Possible race conditions due to rapid container scaling
- Lingering endpoints in a given HCN namespace, potentially not correctly deleted by the CNI
- Nonexistent HCN network namespace trying to be deleted
Environment
- HNS Version: Major: 13, Minor: 3
- Kubernetes Version: v1.29.0
- OS Version: Windows Server 2022 Datacenter 10.0.20348.2340
- Containerd Version: v1.6.28
- CNI: aws-vpc-bridge (L2 bridge networking mode)
- Cloud Platform: Amazon EKS
Similar Issues
These issues cite the same error code might benefit from this issue being responded to
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.