[DOCS] Air Gap Install / Local Repository Mirroring incomplete - RCLONE Command fails to update Hashes + Release / Release.gpg / InRelease Files
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
The Setup Instructions for Air Gap Environments using rclone works for the most Part.
However, for some Reason, the following Files (at least for Debian Repository) fail to properly get updated:
*.md5*.sha1*.sha256ReleaseRelease.gpgInRelease
If the Client does NOT purge the relevant files in /var/lib/apt/lists then this results in a silent Failure and missed Updates.
Right now 3008.2, 3008.1, 3008.0 are available for Download but are not listed because the local Repository never updated due to these old Files (listed above), resulting in apt to still use the ones that "made Sense" (to it at least), instead of Issuing a Warning about the newest Packages List failing to verify Hashes.
root@ubuntu:/home/user# apt policy salt-minion
salt-minion:
Installeret: 3008.0~rc4
Kandidat: 3008.0~rc4
Versionstabel:
*** 3008.0~rc4 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
100 /var/lib/dpkg/status
3008.0~rc3 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3008.0~rc2 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3008.0~rc1 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.14 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.13 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.12 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.11 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.10 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.9 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.8 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.7 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.6 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.5 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.4 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.3 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.2 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.1 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.0 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.25 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.24 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.23 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.22 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.21 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.20 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.19 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.18 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.17 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.16 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.15 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.14 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.13 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.12 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.11 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.10 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.9 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.8 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.7 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.6 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.5 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.4 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.3 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.2 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.1 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.0 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
Try to load new Repository Metadata:
root@ubuntu:/home/user# rm /var/lib/apt/lists/salt.deb.MYDOMAIN.TLD_dists_stable_main_binary-amd64_Packages
root@ubuntu:/home/user# rm /var/lib/apt/lists/salt.deb.MYDOMAIN.TLD_dists_stable_InRelease
root@ubuntu:/home/user# apt update
Hit:1 http://archive.ubuntu.com/ubuntu noble InRelease
Hit:3 http://archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:5 http://security.ubuntu.com/ubuntu noble-security InRelease
Hit:7 http://archive.ubuntu.com/ubuntu noble-backports InRelease
Get:20 https://salt.deb.MYDOMAIN.TLD stable InRelease [3.417 B]
Ign:28 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
Get:28 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages [144 kB]
Err:28 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
File has unexpected size (161260 != 143530). Mirror sync in progress? [IP: fdXX:XXXX:XXXX:1::1:35 443]
Hashes of expected file:
- Filesize:143530 [weak]
- SHA256:8e2b63afa9baf712a02668cf7cf60401ca25af14d38464557dac078dd792749d
- SHA1:17d03bd1701ff651604ae8fa3a0b5b13b9515cdc [weak]
- MD5Sum:82bf6fe9f37f57dcdddf96c492da03b8 [weak]
Release file created at: Fri, 15 May 2026 17:37:43 +0000
Fetched 139 kB in 3s (41,2 kB/s)
Reading package lists... Done
E: Failed to fetch https://salt.deb.MYDOMAIN.TLD/dists/stable/main/binary-amd64/Packages.bz2 File has unexpected size (161260 != 143530). Mirror sync in progress? [IP: fdXX:XXXX:XXXX:1::1:35 443]
Hashes of expected file:
- Filesize:143530 [weak]
- SHA256:8e2b63afa9baf712a02668cf7cf60401ca25af14d38464557dac078dd792749d
- SHA1:17d03bd1701ff651604ae8fa3a0b5b13b9515cdc [weak]
- MD5Sum:82bf6fe9f37f57dcdddf96c492da03b8 [weak]
Release file created at: Fri, 15 May 2026 17:37:43 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.
root@ubuntu:/home/user# apt update
Hit:1 http://archive.ubuntu.com/ubuntu noble InRelease
Get:3 http://archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
Get:14 https://salt.deb.MYDOMAIN.TLD stable InRelease [3.417 B]
Ign:22 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
Get:22 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages [144 kB]
Err:22 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
File has unexpected size (161260 != 143530). Mirror sync in progress? [IP: fdXX:XXXX:XXXX:1::1:35 443]
Hashes of expected file:
- Filesize:143530 [weak]
- SHA256:8e2b63afa9baf712a02668cf7cf60401ca25af14d38464557dac078dd792749d
- SHA1:17d03bd1701ff651604ae8fa3a0b5b13b9515cdc [weak]
- MD5Sum:82bf6fe9f37f57dcdddf96c492da03b8 [weak]
Release file created at: Fri, 15 May 2026 17:37:43 +0000
Hit:23 http://security.ubuntu.com/ubuntu noble-security InRelease
Hit:24 http://archive.ubuntu.com/ubuntu noble-backports InRelease
Get:29 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages [1.073 kB]
Get:30 http://archive.ubuntu.com/ubuntu noble-updates/main i386 Packages [510 kB]
Get:31 http://archive.ubuntu.com/ubuntu noble-updates/restricted i386 Packages [28,8 kB]
Get:32 http://archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Packages [1.196 kB]
Get:33 http://archive.ubuntu.com/ubuntu noble-updates/universe i386 Packages [1.129 kB]
Get:34 http://archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages [1.659 kB]
Reading package lists... Done
root@ubuntu:/home/user# apt update
Hit:3 http://archive.ubuntu.com/ubuntu noble InRelease
Hit:5 http://archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:7 http://security.ubuntu.com/ubuntu noble-security InRelease
Hit:9 http://archive.ubuntu.com/ubuntu noble-backports InRelease
Get:20 https://salt.deb.MYDOMAIN.TLD stable InRelease [3.417 B]
Ign:28 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
Get:28 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages [144 kB]
Err:28 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
File has unexpected size (161260 != 143530). Mirror sync in progress? [IP: 192.168.1.35 443]
Hashes of expected file:
- Filesize:143530 [weak]
- SHA256:8e2b63afa9baf712a02668cf7cf60401ca25af14d38464557dac078dd792749d
- SHA1:17d03bd1701ff651604ae8fa3a0b5b13b9515cdc [weak]
- MD5Sum:82bf6fe9f37f57dcdddf96c492da03b8 [weak]
Release file created at: Fri, 15 May 2026 17:37:43 +0000
Reading package lists... Done
root@ubuntu:/home/user# rm /var/lib/apt/lists/partial/salt.deb.MYDOMAIN.TLD_dists_stable_main_binary-amd64_Packages.bz2.FAILED
root@ubuntu:/home/user# rm /var/lib/apt/lists/partial/salt.deb.MYDOMAIN.TLD_dists_stable_InRelease
root@ubuntu:/home/user# apt update
Hit:1 http://archive.ubuntu.com/ubuntu noble InRelease
Hit:2 http://archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:4 http://security.ubuntu.com/ubuntu noble-security InRelease
Hit:5 http://archive.ubuntu.com/ubuntu noble-backports InRelease
Get:20 https://salt.deb.MYDOMAIN.TLD stable InRelease [3.417 B]
Ign:28 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
Get:28 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages [144 kB]
Err:28 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
File has unexpected size (161260 != 143530). Mirror sync in progress? [IP: fdXX:XXXX:XXXX:1::1:35 443]
Hashes of expected file:
- Filesize:143530 [weak]
- SHA256:8e2b63afa9baf712a02668cf7cf60401ca25af14d38464557dac078dd792749d
- SHA1:17d03bd1701ff651604ae8fa3a0b5b13b9515cdc [weak]
- MD5Sum:82bf6fe9f37f57dcdddf96c492da03b8 [weak]
Release file created at: Fri, 15 May 2026 17:37:43 +0000
Reading package lists... Done
Suggested Fix
There might be missing Options for the rclone Command or something else, I'm not entirely sure. What I know is that those Files are not getting updated (May 15th instead of July 01st):
[mirror@salt-mirror binary-amd64]$ ls -la /data/salt-mirror/saltproject/deb/dists/stable/main/binary-arm64/
total 604
drwxr-xr-x 3 mirror mirror 15 Jul 1 21:47 .
drwxr-xr-x 5 mirror mirror 5 Jul 1 21:47 ..
-rw-r--r-- 1 mirror mirror 773951 Jul 1 21:47 Packages
-rw-r--r-- 1 mirror mirror 161260 Jul 1 21:47 Packages.bz2
-rw-r--r-- 1 mirror mirror 32 May 15 19:37 Packages.bz2.md5
-rw-r--r-- 1 mirror mirror 40 May 15 19:37 Packages.bz2.sha1
-rw-r--r-- 1 mirror mirror 64 May 15 19:37 Packages.bz2.sha256
-rw-r--r-- 1 mirror mirror 168670 Jul 1 21:47 Packages.gz
-rw-r--r-- 1 mirror mirror 32 May 15 19:37 Packages.gz.md5
-rw-r--r-- 1 mirror mirror 40 May 15 19:37 Packages.gz.sha1
-rw-r--r-- 1 mirror mirror 64 May 15 19:37 Packages.gz.sha256
-rw-r--r-- 1 mirror mirror 32 May 15 19:37 Packages.md5
-rw-r--r-- 1 mirror mirror 40 May 15 19:37 Packages.sha1
-rw-r--r-- 1 mirror mirror 64 May 15 19:37 Packages.sha256
drwxr-xr-x 5 mirror mirror 5 Jul 1 21:47 by-hash
[mirror@salt-mirror scripts]$ ls -la /data/salt-mirror/saltproject/deb/dists/stable/
total 41
drwxr-xr-x 3 mirror mirror 15 Jul 1 21:47 .
drwxr-xr-x 3 mirror mirror 3 Oct 25 2024 ..
-rw-r--r-- 1 mirror mirror 3417 May 15 19:37 InRelease
-rw-r--r-- 1 mirror mirror 32 Jul 1 21:47 InRelease.md5
-rw-r--r-- 1 mirror mirror 40 Jul 1 21:47 InRelease.sha1
-rw-r--r-- 1 mirror mirror 64 Jul 1 21:47 InRelease.sha256
-rw-r--r-- 1 mirror mirror 2718 May 15 19:37 Release
-rw-r--r-- 1 mirror mirror 650 May 15 19:37 Release.gpg
-rw-r--r-- 1 mirror mirror 32 Jul 1 21:47 Release.gpg.md5
-rw-r--r-- 1 mirror mirror 40 Jul 1 21:47 Release.gpg.sha1
-rw-r--r-- 1 mirror mirror 64 Jul 1 21:47 Release.gpg.sha256
-rw-r--r-- 1 mirror mirror 32 Jul 1 21:47 Release.md5
-rw-r--r-- 1 mirror mirror 40 Jul 1 21:47 Release.sha1
-rw-r--r-- 1 mirror mirror 64 Jul 1 21:47 Release.sha256
drwxr-xr-x 5 mirror mirror 5 Jul 1 21:47 main
Current Workaround is to delete the affected Files for Debian Repository at every Repository Synchronization.
Workaround Script:
#!/bin/bash
## Target Folder
TARGET_FOLDER="/data/salt-mirror"
# Remove existing Hashes
mapfile -t hash_files < <( find "${TARGET_FOLDER}/saltproject/deb/dists" -iwholename "*/*.sha1" -o -iwholename "*/*.sha256" -o -iwholename "*/*.md5" -o -iwholename "*/Release" -o -iwholename "*/InRelease" -o -iwholename "*/Release.gpg" )
# Loop over Hashes
echo "Begin Removal of confusing Hash Files"
for hash_file in "${hash_files[@]}"
do
# Echo
echo "Remove File ${hash_file}"
# Perform Removal
rm "${hash_file}"
done
echo "Finished Removal of confusing Hash Files"
## Mirror RPM Repository
echo "Mirror RPM Repository"
RCLONE_WEBDAV_URL=https://packages.broadcom.com/artifactory/ RCLONE_WEBDAV_VENDOR=other rclone sync --fast-list --use-server-modtime -v :webdav:saltproject-rpm/ "${TARGET_FOLDER}/saltproject/rpm/"
## Mirror DEB Repository
echo "Mirror DEB Repository"
RCLONE_WEBDAV_URL=https://packages.broadcom.com/artifactory/ RCLONE_WEBDAV_VENDOR=other rclone sync --fast-list --use-server-modtime -v :webdav:saltproject-deb/ "${TARGET_FOLDER}/saltproject/deb/"
After that on the Client Device the new Packages get properly listed and can be upgraded:
root@ubuntu:/home/user# apt policy salt-minion
salt-minion:
Installeret: 3008.0~rc4
Kandidat: 3008.2
Versionstabel:
3008.2 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3008.1 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3008.0 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
*** 3008.0~rc4 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
100 /var/lib/dpkg/status
3008.0~rc3 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3008.0~rc2 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3008.0~rc1 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.14 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.13 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.12 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.11 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.10 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.9 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.8 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.7 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.6 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.5 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.4 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.3 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.2 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.1 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3007.0 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.27 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.26 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.25 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.24 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.23 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.22 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.21 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.20 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.19 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.18 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.17 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.16 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.15 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.14 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.13 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.12 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.11 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.10 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.9 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.8 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.7 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.6 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.5 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.4 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.3 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.2 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.1 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
3006.0 500
500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
Type of documentation
Salt Documentation.
Location or format of documentation
Air Gap Install
Additional context
Add any other context or screenshots about the feature request here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue does not name a documentation file; first locate the Air Gap Environments and rclone setup instructions, then reproduce the repository mirror update with the listed Debian metadata files. Done means the instructions reliably refresh checksums and Release metadata so apt update detects the newest package lists without manually purging /var/lib/apt/lists.
Written by the indexing model from the issue text.
Assessment
- Domain
- devops, documentation
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 48/100