Azure / Azure/azure-functions-host
zipdeploy with WEBSITE_RUN_FROM_PACKAGE=1 removes the active package before validating the replacement — a failed deploy leaves the host unable to start
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 38
Description
### Summary
With `WEBSITE_RUN_FROM_PACKAGE=1`, a **failed** `zipdeploy` can leave the Function App unable to start, because the previously-active package is removed from `SitePackages` before the replacement has been validated. A deploy that fails should be non-destructive; this one turned into a 90-minute production outage.
### Environment
- Windows Function App, Elastic Premium plan
- `WEBSITE_RUN_FROM_PACKAGE=1` (local `SitePackages`, not a blob URL)
- Deployed via Kudu `zipdeploy` (through `Azure/functions-action`)
- ~218 MB deployment package (Node, large `node_modules` plus bundled binaries)
### What happened
Three consecutive deploys failed with `HTTP 500` from `/api/zipdeploy`. After all three, the app crash-looped on:
```
Shutting down host due to presence of
C:\home\site\wwwroot\FAILED TO INITIALIZE RUN FROM PACKAGE.txt
File content: Run From Package Initialization failed.
```
Inspecting `data/SitePackages` explained why:
```
packagename.txt (mtime = the LAST SUCCESSFUL deploy) -> ".zip"
SitePackages/ actual contents:
.zip 49,708,689 bytes truncated
.zip 50,078,965 bytes truncated
.zip 50,250,385 bytes truncated
packagename.txt 18 bytes
^^ ".zip" — the package the pointer names — is ABSENT
```
So the pointer still referenced the package that had been serving successfully, but that file was gone, while none of the three replacements ever completed. Each partial zip has a valid `PK\003\004` local header and **no end-of-central-directory record** — i.e. cut off mid-write.
The app kept running for ~77 minutes after the first failed deploy (the already-mounted package stayed live), then a worker recycle tried to re-mount, found nothing, and wrote the failure marker. Every restart from then on hit the same marker.
### Why the uploads failed (contributing, but not the defect being reported)
Writes through the Kudu path were extremely slow at the time. Measured against **the same file share, from the same client, minutes apart**:
| Path | Payload | Throughput | Result |
|---|---|---|---|
| Kudu VFS | 5 MB | 1.06 MB/s | 201 |
| Kudu VFS | 20 MB | 0.83 MB/s | 201 |
| Kudu VFS | 218 MB | ~1 MB/s | 502, truncated |
| Azure Files (direct, same share) | 218 MB | **~38 MB/s** | complete |
Small writes succeeded, so the path wasn't broken — just ~40× slower than the storage behind it, which meant anything over ~80 MB could not finish inside the SCM gateway's ~90s limit. Azure Resource Health reported the storage account healthy throughout.
That slowness is presumably environment-specific and I'm not asking for it to be diagnosed here. **The defect is that a deploy which cannot complete destroys the working artifact.**
### Expected behaviour
A `zipdeploy` that fails to land its new package should leave the app running the package it already had. Concretely: don't prune/remove the currently-referenced package until the replacement has been written completely, validated, and `packagename.txt` flipped.
### Actual behaviour
The active package is gone, `packagename.txt` still names it, and the host has nothing to mount on its next recycle. There is also no supported recovery from that state short of manually writing a package to the share and repointing `packagename.txt`, which is what I ended up doing.
### Notes
- Nothing was dead-lettered or logged as an error at the moment of removal — the loss is silent until the next worker recycle, which can be over an hour later. That delay makes the deploy failure and the outage look unrelated.
- Switching this app to blob-backed `WEBSITE_RUN_FROM_PACKAGE=` removes the exposure entirely, since each deploy is then an immutable blob and the swap is a single setting flip with the previous package still intact. That's the workaround I've adopted, and it may be worth documenting as the recommended pattern for large packages.
Contributor guide
Research direction
Start at the Kudu /api/zipdeploy path and trace how packages are written and pruned in data/SitePackages, including the packagename.txt pointer. Reproduce a failed or truncated upload and verify that the previously referenced package remains present and mountable, while packagename.txt changes only after the replacement is complete and validated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp
- Domain
- backend, cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100