Azure / Azure/azure-sdk-tools

Fix stress parallel deployment deadlock issue

Open
#8,400 0 comments 0 reactions 0 assignees View on GitHub
Central-EngSys Stress
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 1h
Merged PRs (30d)
143

Description

When using the parallel feature (indexed jobs) in stress, we gate ARM deploys on pod 0 completing deployment to avoid race conditions. There is a deadlock issue where many pods are deployed exhausting our node resource quota. If pod 0 does not get scheduled, then pods 1+n will use up all quota polling for pod 0 completion, but pod 0 will never be scheduled.

Contributor guide

Open the contributing guide

Research direction

Locate the stress parallel indexed-job deployment code that gates ARM deploys on pod 0, then inspect how pods poll for completion and consume node quota. Reproduce the case where pod 0 is unscheduled; done means the remaining pods cannot exhaust the quota while waiting and the deployment can recover or terminate clearly.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.