hypothesis / hypothesis/lms

Possible race condition with asset serving during deployments

Open
#2,799 8 comments 0 reactions 1 assignee Claimed by @robertknight View on GitHub
Dominant language
Python
Stars
53
Forks
16
Avg merge
14d 5h
Merged PRs (30d)
14

Description

When a new deployment happens we do a rolling update with an additional batch. I need to confirm but if it is possible that during a deployment a page load can be serviced by both the old and new versions of the app then we could have a situation where an old asset gets cached under a new URL. This might explain an issue [seen by a user](https://hypothes-is.slack.com/archives/C2BLQDKHA/p1623242191165700) where we deployed a bug fix but the user still didn't see the fix some hours later.

The problem scenario is:

1. LMS page request hits instance running new version of app. This serves up a response with new asset URLs (ie. that have the latest cache-busting query string at the end)
2. Browser requests asset with new URL. Cloudflare connects to LMS app and hits instance running old version of app. Since the app ignores the query string when serving the asset, it might still serve the request even if the query string is wrong.
3. Cloudflare edge location caches the old version of the asset under the new URL

To re-iterate, the main condition we need to check here is whether the current deployment process makes it possible that during a single page load requests may go to both the old and new versions of the app during the deployment window.

If this problem does exist then it could affect other apps too. Usage is way up over past years, so it is possible that the issue has long existed but simply been rare.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.