Remote state writes have no concurrency control, so concurrent operations silently lose data
- Dominant language
- Go
- Stars
- 569
- Forks
- 364
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 136
Description
### Problem
The `AzureBlobStorage` remote state backend performs read-modify-write on environment state with no optimistic concurrency check and no lease. Two operations running against the same environment at the same time will silently discard one set of writes. There is no error, no warning, and no way for either caller to detect that it happened.
### Evidence
`cli/azd/pkg/azsdk/storage/storage_blob_client.go`:
```go
_, err := bc.client.UploadStream(ctx, bc.config.ContainerName, blobPath, reader, nil)
```
The final argument is the options struct. Passing `nil` means no `AccessConditions`, so no `If-Match` ETag precondition and no lease ID. The file contains no reference to `ETag`, `IfMatch`, `AccessConditions`, or `Lease`. Neither does `cli/azd/pkg/environment/storage_blob_data_store.go`.
Every command that writes environment values goes through this path, including `azd env set`, and the output-persisting steps of `azd provision` and `azd deploy`.
### Impact
Last writer wins, and the loser is not told. Concretely:
- Two concurrent deployments to one environment can interleave such that infrastructure outputs written by one are overwritten by state read before those outputs existed.
- Any hook that records state (deployed artifact identifiers, migration or schema markers, feature flags) can have its write erased by an unrelated concurrent operation.
- The corruption is silent and persists, so the next read returns a state that never existed as a consistent snapshot. Recovery requires knowing it happened.
Remote state is presented as the mechanism that lets state travel with an environment across machines and automation. Multiple writers are therefore the expected case, not an edge case, and this is precisely the configuration in which the backend is unsafe.
### Proposed requirement
Remote state writes must be safe under concurrency.
1. Capture the blob ETag on read and send it as an `If-Match` precondition on write. On a 412, fail with an explicit conflict error naming the environment. Never overwrite blindly.
2. For multi-step mutating commands (`provision`, `deploy`, `up`, `down`), acquire a blob lease on the environment's state for the duration and release it on completion or failure.
3. A caller blocked by a conflict or an existing lease must get an actionable message identifying the environment and, where available, the holder.
### Acceptance criteria
- [ ] Blob writes send an `If-Match` ETag precondition
- [ ] ETag mismatch surfaces a distinct conflict error rather than succeeding
- [ ] Mutating commands hold a lease on environment state for their duration
- [ ] Leases are released on success, failure, and interrupt
- [ ] A stale or abandoned lease has a documented expiry and recovery path
- [ ] Test covers two concurrent writers and asserts neither loses data silently
Contributor guide
Research direction
Start with cli/azd/pkg/azsdk/storage/storage_blob_client.go and cli/azd/pkg/environment/storage_blob_data_store.go, then trace the state-writing paths used by azd env set, azd provision, and azd deploy. Review the Azure Blob conditional-write and lease APIs, and define tests for concurrent writers, conflict reporting, lease cleanup, and stale-lease recovery. Done means all acceptance criteria pass without silent data loss.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, go
- Domain
- cli, cloud, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100