Bot behavior - 404 handling

Background

One of the activities of the bots is to scan the repo to ensure that installer hashes match and that URL's are not 404. This is fairly complex behavior, so to level-set and ensure that I am of the correct understanding, please let me know if I am missing any part of the intended functionality here or if it has been changed since the original implementation

Expected Behavior

1. Scan the installer URL's for the latest version of a package
   • If a hash mismatch is detected, attempt an update
   • If no mismatch is detected, continue
   • If the URL is 404, create an issue here in winget-pkgs
2. Scan the installer URL's for all other versions of a package
   • If a hash mismatch is detected, remove the version
   • If no mismatch is detected, continue
   • If the URL is 404, remove the version

With the above behavior, the latest version of a package will never be removed automatically, and community members will be notified that an update is required, decreasing the downtime for packages which are 404 and not able to be updated. Additionally, older versions of packages which are no longer working would be removed, ensuring the most accurate version list is available.

Discovery

I have a custom function I run which gets a URL response. This is the same function which is used in the YamlCreate script to determine if a URL is valid before accepting it. With this, I decided to do a test scan of the packages just out of curiosity, and found several 404's which are not the latest version.

Reproduction

These reproduction steps were created on PowerShell 5.1.19041.1682 and have not been tested for newer versions

1. Open PowerShell and install the powershell-yaml module - Install-Module -Name powershell-yaml
2. Clone the winget-pkgs repository and navigate to the directory in PowerShell
3. Collect all the installer files - $files = gci -recurse -File -Filter '*.installer.yaml'
4. Set up the Get-UrlResponse function by copying and pasting the code from the gist into the PowerShell session and running it
5. Loop across all the installers in all the manifests, testing the URL for a 404

$output = @(); $files | ForEach-Object {
    Write-Progress -Activity 'Scanning Manifests' -CurrentOperation $_.FullName -Id 1001
    $yaml = Get-Content $_.FullName | ConvertFrom-Yaml;
    foreach ($installer in $yaml.Installers) {
        Write-Progress -Activity 'Scanning Installer URL' -CurrentOperation $installer.InstallerUrl -Id 1002
        $resp = Get-UrlResponse $installer.InstallerUrl
        if ($resp.ResponseCode -eq 404){
            $output += $_.FullName
            Write-Host "404 at $($resp.Url)"
        }
    }
}

When doing this, I found that packages like Alibaba.Yuque had many versions which were 404. The latest version was not 404, but a majority of the older versions were.

404 at https://github.com/TDesktop-x64/tdesktop/releases/download/v1.0.38/64Gram-setup-x64-1.0.38.exe
404 at https://github.com/TDesktop-x64/tdesktop/releases/download/v1.0.38/64Gram-setup.1.0.38.exe
404 at https://download.effie.co/effie/effie_setup_2.1.5.exe
404 at https://download.effie.co/effie/effie_setup_2.1.7.exe
404 at https://8gadgetpack.net/dl_340/8GadgetPackSetup.msi
404 at https://aimp.ru/files/windows/builds/aimp_5.03.2394_w32.exe
404 at https://yunpan.aliyun.com/downloads/apps/desktop/aDrive-2.2.7.exe
404 at https://yunpan.aliyun.com/downloads/apps/desktop/aDrive-2.2.8.exe
404 at https://yunpan.aliyun.com/downloads/apps/desktop/aDrive-2.3.3.exe
404 at https://yunpan.aliyun.com/downloads/apps/desktop/aDrive-2.3.4.exe
404 at https://yunpan.aliyun.com/downloads/apps/desktop/aDrive-2.3.5.exe
404 at https://yunpan.aliyun.com/downloads/apps/desktop/aDrive-2.3.6.exe
404 at https://dtapp-pub.dingtalk.com/dingtalk-desktop/win_installer/Release/DingTalk_v6.3.25.1209106.exe
404 at https://dtapp-pub.dingtalk.com/dingtalk-desktop/win_installer/Release/DingTalk_v6.3.25.1219101.exe
404 at https://app.nlark.com/yuque-desktop/Yuque-0.10.2.exe
404 at https://app.nlark.com/yuque-desktop/Yuque-0.10.2.exe
404 at https://app.nlark.com/yuque-desktop/Yuque-0.8.13.exe
404 at https://app.nlark.com/yuque-desktop/Yuque-0.8.13.exe
404 at https://app.nlark.com/yuque-desktop/Yuque-0.9.31.exe
404 at https://app.nlark.com/yuque-desktop/Yuque-0.9.31.exe
404 at https://app.nlark.com/yuque-desktop/Yuque-1.0.3.exe
404 at https://app.nlark.com/yuque-desktop/Yuque-1.0.3.exe
404 at https://download.ascension-patch.gg/update/ascension-setup-79.exe
404 at https://cdn.axis.com/ftp/pub_soft/cam_srv/cam_station/cam_station_preview/0_2130_58/AXISCameraStationSetup.exe
404 at https://issuepcdn.baidupcs.com/issue/netdisk/yunguanjia/BaiduNetdisk_7.17.5.19.exe
404 at https://issuepcdn.baidupcs.com/issue/netdisk/yunguanjia/BaiduNetdisk_7.17.6.2.exe
404 at https://issuepcdn.baidupcs.com/issue/netdisk/yunguanjia/BaiduNetdisk_7.18.0.15.exe

Related Items

• microsoft/winget-cli#1900

Start with the YamlCreate script and the Get-UrlResponse function described in the reproduction, then review microsoft/winget-cli#1900 for related context. A complete change would scan latest and older installer URLs, preserve the latest version on 404 or hash mismatch, and notify or remove versions according to the stated behavior.