Azure / Azure/app-service-linux-docs

Stale output.tar.zst from prior deploy not cleaned up; runtime extracts orphan over current output.tar.gz

Open
#263 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
150
Forks
90
Avg merge
2m
Merged PRs (30d)
1

Description

## Summary

On Linux Python App Service, when consecutive deploys produce different compression formats (`output.tar.zst` from one, `output.tar.gz` from a later one), both files persist in `/home/site/wwwroot/`. The Oryx runtime startup script extracts the OLDER `output.tar.zst` instead of the newer `output.tar.gz`, causing production to serve stale code with no error indication.

## Environment

- App Service: `wapd-image-finder` (Linux, Python 3.11)
- Image tag: `appsvc/python:3.11_20260203.2.tuxprod`
- Oryx Version: `0.2.20251017.2`
- Date observed: 2026-05-10 UTC
- Region: East US

## Reproduction sequence

1. Deploy A produces `output.tar.zst` (zstd-compressed) -- in our case a deploy attempt where `COMPRESS_DESTINATION_DIR=false` was set as an app setting but ignored by Oryx (separate issue, see microsoft/Oryx#2921).
2. Deploy B (later, via GitHub Actions OneDeploy) produces `output.tar.gz` (gzip).
3. Both files now sit in `/home/site/wwwroot/`. Microsoft does not delete the older one.
4. Container restarts. Runtime startup script extracts `output.tar.zst` (older) rather than `output.tar.gz` (current).
5. Production serves the OLDER deploy's source/static-assets.

## Evidence

```
$ ls -la /home/site/wwwroot/
-rwxrwxrwx 1 nobody nogroup 261 May 10 04:47 oryx-manifest.toml
-rwxrwxrwx 1 nobody nogroup 289966627 May 10 04:47 output.tar.gz <- current deploy
-rwxrwxrwx 1 nobody nogroup 277256844 May 10 02:23 output.tar.zst <- orphan from prior deploy

$ tar -tzf /home/site/wwwroot/output.tar.gz | grep assets/index
./frontend/dist/assets/index-w8ghkh8t.css
./frontend/dist/assets/index-DNf7ivpY.js <- new bundle hash, 404 in production

$ zstd -d -c /home/site/wwwroot/output.tar.zst | tar -t | grep assets/index
./frontend/dist/assets/index-BRgwFiec.js <- old bundle hash, 200 in production
./frontend/dist/assets/index-CjFrALqX.css
```

`oryx-manifest.toml` from the latest deploy says `CompressDestinationDir="true"`. The startup script appears to scan for any `output.tar.*` file rather than reading the manifest's intended format, picking the older `.zst` over the newer `.gz`.

## Workaround

Manually `rm /home/site/wwwroot/output.tar.zst` via Kudu shell, then `az webapp restart`. After the restart, runtime extracts only `output.tar.gz` and production serves the current deploy.

## Asks

1. Oryx runtime startup script should either (a) trust `oryx-manifest.toml`'s declared format and only look for the matching tarball, or (b) prefer the newest `output.tar.*` file by mtime.
2. Oryx build phase should clean up prior `output.tar.*` artifacts in `wwwroot` so only the newest exists after each deploy.
3. Fix `COMPRESS_DESTINATION_DIR=false` honoring (currently ignored by Oryx, tracked in microsoft/Oryx#2921). With that env var working, customers wouldn't end up with mixed-format tarballs in the first place.

## Related Microsoft issues we've filed today

- microsoft/Oryx#2921 -- Container panic when build writes `.zst` but runtime expects `.gz`
- Azure/app-service-linux-docs#262 -- OneDeploy 503 Service Unavailable post-auth on Linux Python deploys

This is the third distinct App Service Linux regression we've hit in a 24h window. Would appreciate triage.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.