Every push to the main branch triggers an upload to download.pytorch.org

🐛 Describe the bug

Recently while testing torchbench with @xuzhao9 , we seem to encounter more cases of date misalignment between Torch and the domain library (e.g. TorchVision).
e.g. the following two machines (one from west and the other central) were downloading from the same link

wget https://download.pytorch.org/whl/nightly/cu117/torchvision-0.16.0.dev20230331%2Bcu117-cp310-cp310-linux_x86_64.whl

weiwangmeta@machine1:~ $ pkginfo -f requires_dist ./torchvision-0.16.0.dev20230331+cu117-cp310-cp310-linux_x86_64.whl 
requires_dist: ['numpy', 'requests', 'torch (==2.1.0.dev20230331)', 'pillow (!=8.3.*,>=5.3.0)', "scipy ; extra == 'scipy'"]

weiwangmeta@machine2:~ $ pkginfo -f requires_dist ./torchvision-0.16.0.dev20230331+cu117-cp310-cp310-linux_x86_64.whl  
requires_dist: ['numpy', 'requests', 'torch (==2.1.0.dev20230330)', 'pillow (!=8.3.*,>=5.3.0)', "scipy ; extra == 'scipy'"]

Clearly, this is a caching issue where we need to flush the cache (cc @atalman). But why are we seeming to have different vision binaries generated today?

Digging deeper into it, I see a total of three versions/binaries with the same name were uploaded. The first two were too early to see torch (==2.1.0.dev20230331), making them depend on torch(==2.1.0.dev20230330). Only the 3rd one, which is from the nightly push, is the expected correct binary.

So digging deeper, it seems the build-wheels-linux job ran on every commit of main branch.

This is more a test-infra issue, but logging it in this repo since that is where we first notice this problem.

Versions

At least 03312023 nightly.