saltstack / saltstack/salt

[DOCS] Air Gap Install / Local Repository Mirroring incomplete - RCLONE Command fails to update Hashes + Release / Release.gpg / InRelease Files

Open
#69,652 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

documentation
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
The Setup Instructions for Air Gap Environments using rclone works for the most Part.

However, for some Reason, the following Files (at least for Debian Repository) fail to properly get updated:

  • *.md5
  • *.sha1
  • *.sha256
  • Release
  • Release.gpg
  • InRelease

If the Client does NOT purge the relevant files in /var/lib/apt/lists then this results in a silent Failure and missed Updates.

Right now 3008.2, 3008.1, 3008.0 are available for Download but are not listed because the local Repository never updated due to these old Files (listed above), resulting in apt to still use the ones that "made Sense" (to it at least), instead of Issuing a Warning about the newest Packages List failing to verify Hashes.

root@ubuntu:/home/user# apt policy salt-minion
salt-minion:
  Installeret: 3008.0~rc4
  Kandidat:    3008.0~rc4
  Versionstabel:
 *** 3008.0~rc4 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
        100 /var/lib/dpkg/status
     3008.0~rc3 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3008.0~rc2 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3008.0~rc1 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.14 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.13 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.12 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.11 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.10 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.9 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.8 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.7 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.6 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.5 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.4 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.3 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.2 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.1 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.0 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.25 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.24 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.23 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.22 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.21 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.20 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.19 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.18 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.17 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.16 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.15 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.14 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.13 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.12 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.11 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.10 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.9 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.8 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.7 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.6 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.5 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.4 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.3 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.2 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.1 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.0 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages

Try to load new Repository Metadata:

root@ubuntu:/home/user# rm /var/lib/apt/lists/salt.deb.MYDOMAIN.TLD_dists_stable_main_binary-amd64_Packages
root@ubuntu:/home/user# rm /var/lib/apt/lists/salt.deb.MYDOMAIN.TLD_dists_stable_InRelease

root@ubuntu:/home/user# apt update
Hit:1 http://archive.ubuntu.com/ubuntu noble InRelease
Hit:3 http://archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:5 http://security.ubuntu.com/ubuntu noble-security InRelease
Hit:7 http://archive.ubuntu.com/ubuntu noble-backports InRelease
Get:20 https://salt.deb.MYDOMAIN.TLD stable InRelease [3.417 B]
Ign:28 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
Get:28 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages [144 kB]
Err:28 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
  File has unexpected size (161260 != 143530). Mirror sync in progress? [IP: fdXX:XXXX:XXXX:1::1:35 443]
  Hashes of expected file:
   - Filesize:143530 [weak]
   - SHA256:8e2b63afa9baf712a02668cf7cf60401ca25af14d38464557dac078dd792749d
   - SHA1:17d03bd1701ff651604ae8fa3a0b5b13b9515cdc [weak]
   - MD5Sum:82bf6fe9f37f57dcdddf96c492da03b8 [weak]
  Release file created at: Fri, 15 May 2026 17:37:43 +0000
Fetched 139 kB in 3s (41,2 kB/s)
Reading package lists... Done
E: Failed to fetch https://salt.deb.MYDOMAIN.TLD/dists/stable/main/binary-amd64/Packages.bz2 File has unexpected size (161260 != 143530). Mirror sync in progress? [IP: fdXX:XXXX:XXXX:1::1:35 443]
   Hashes of expected file:
    - Filesize:143530 [weak]
    - SHA256:8e2b63afa9baf712a02668cf7cf60401ca25af14d38464557dac078dd792749d
    - SHA1:17d03bd1701ff651604ae8fa3a0b5b13b9515cdc [weak]
    - MD5Sum:82bf6fe9f37f57dcdddf96c492da03b8 [weak]
   Release file created at: Fri, 15 May 2026 17:37:43 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.

root@ubuntu:/home/user# apt update
Hit:1 http://archive.ubuntu.com/ubuntu noble InRelease
Get:3 http://archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
Get:14 https://salt.deb.MYDOMAIN.TLD stable InRelease [3.417 B]
Ign:22 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
Get:22 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages [144 kB]
Err:22 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
  File has unexpected size (161260 != 143530). Mirror sync in progress? [IP: fdXX:XXXX:XXXX:1::1:35 443]
  Hashes of expected file:
   - Filesize:143530 [weak]
   - SHA256:8e2b63afa9baf712a02668cf7cf60401ca25af14d38464557dac078dd792749d
   - SHA1:17d03bd1701ff651604ae8fa3a0b5b13b9515cdc [weak]
   - MD5Sum:82bf6fe9f37f57dcdddf96c492da03b8 [weak]
  Release file created at: Fri, 15 May 2026 17:37:43 +0000
Hit:23 http://security.ubuntu.com/ubuntu noble-security InRelease
Hit:24 http://archive.ubuntu.com/ubuntu noble-backports InRelease
Get:29 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages [1.073 kB]
Get:30 http://archive.ubuntu.com/ubuntu noble-updates/main i386 Packages [510 kB]
Get:31 http://archive.ubuntu.com/ubuntu noble-updates/restricted i386 Packages [28,8 kB]
Get:32 http://archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Packages [1.196 kB]
Get:33 http://archive.ubuntu.com/ubuntu noble-updates/universe i386 Packages [1.129 kB]
Get:34 http://archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages [1.659 kB]
Reading package lists... Done

root@ubuntu:/home/user# apt update
Hit:3 http://archive.ubuntu.com/ubuntu noble InRelease
Hit:5 http://archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:7 http://security.ubuntu.com/ubuntu noble-security InRelease   
Hit:9 http://archive.ubuntu.com/ubuntu noble-backports InRelease                  
Get:20 https://salt.deb.MYDOMAIN.TLD stable InRelease [3.417 B]
Ign:28 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
Get:28 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages [144 kB]
Err:28 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
  File has unexpected size (161260 != 143530). Mirror sync in progress? [IP: 192.168.1.35 443]
  Hashes of expected file:
   - Filesize:143530 [weak]
   - SHA256:8e2b63afa9baf712a02668cf7cf60401ca25af14d38464557dac078dd792749d
   - SHA1:17d03bd1701ff651604ae8fa3a0b5b13b9515cdc [weak]
   - MD5Sum:82bf6fe9f37f57dcdddf96c492da03b8 [weak]
  Release file created at: Fri, 15 May 2026 17:37:43 +0000
Reading package lists... Done

root@ubuntu:/home/user# rm /var/lib/apt/lists/partial/salt.deb.MYDOMAIN.TLD_dists_stable_main_binary-amd64_Packages.bz2.FAILED
root@ubuntu:/home/user# rm /var/lib/apt/lists/partial/salt.deb.MYDOMAIN.TLD_dists_stable_InRelease

root@ubuntu:/home/user# apt update
Hit:1 http://archive.ubuntu.com/ubuntu noble InRelease
Hit:2 http://archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:4 http://security.ubuntu.com/ubuntu noble-security InRelease
Hit:5 http://archive.ubuntu.com/ubuntu noble-backports InRelease
Get:20 https://salt.deb.MYDOMAIN.TLD stable InRelease [3.417 B]    
Ign:28 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
Get:28 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages [144 kB]
Err:28 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
  File has unexpected size (161260 != 143530). Mirror sync in progress? [IP: fdXX:XXXX:XXXX:1::1:35 443]
  Hashes of expected file:
   - Filesize:143530 [weak]
   - SHA256:8e2b63afa9baf712a02668cf7cf60401ca25af14d38464557dac078dd792749d
   - SHA1:17d03bd1701ff651604ae8fa3a0b5b13b9515cdc [weak]
   - MD5Sum:82bf6fe9f37f57dcdddf96c492da03b8 [weak]
  Release file created at: Fri, 15 May 2026 17:37:43 +0000
Reading package lists... Done

Suggested Fix
There might be missing Options for the rclone Command or something else, I'm not entirely sure. What I know is that those Files are not getting updated (May 15th instead of July 01st):

[mirror@salt-mirror binary-amd64]$ ls -la /data/salt-mirror/saltproject/deb/dists/stable/main/binary-arm64/
total 604
drwxr-xr-x 3 mirror mirror     15 Jul  1 21:47 .
drwxr-xr-x 5 mirror mirror      5 Jul  1 21:47 ..
-rw-r--r-- 1 mirror mirror 773951 Jul  1 21:47 Packages
-rw-r--r-- 1 mirror mirror 161260 Jul  1 21:47 Packages.bz2
-rw-r--r-- 1 mirror mirror     32 May 15 19:37 Packages.bz2.md5
-rw-r--r-- 1 mirror mirror     40 May 15 19:37 Packages.bz2.sha1
-rw-r--r-- 1 mirror mirror     64 May 15 19:37 Packages.bz2.sha256
-rw-r--r-- 1 mirror mirror 168670 Jul  1 21:47 Packages.gz
-rw-r--r-- 1 mirror mirror     32 May 15 19:37 Packages.gz.md5
-rw-r--r-- 1 mirror mirror     40 May 15 19:37 Packages.gz.sha1
-rw-r--r-- 1 mirror mirror     64 May 15 19:37 Packages.gz.sha256
-rw-r--r-- 1 mirror mirror     32 May 15 19:37 Packages.md5
-rw-r--r-- 1 mirror mirror     40 May 15 19:37 Packages.sha1
-rw-r--r-- 1 mirror mirror     64 May 15 19:37 Packages.sha256
drwxr-xr-x 5 mirror mirror      5 Jul  1 21:47 by-hash

[mirror@salt-mirror scripts]$ ls -la /data/salt-mirror/saltproject/deb/dists/stable/
total 41
drwxr-xr-x 3 mirror mirror   15 Jul  1 21:47 .
drwxr-xr-x 3 mirror mirror    3 Oct 25  2024 ..
-rw-r--r-- 1 mirror mirror 3417 May 15 19:37 InRelease
-rw-r--r-- 1 mirror mirror   32 Jul  1 21:47 InRelease.md5
-rw-r--r-- 1 mirror mirror   40 Jul  1 21:47 InRelease.sha1
-rw-r--r-- 1 mirror mirror   64 Jul  1 21:47 InRelease.sha256
-rw-r--r-- 1 mirror mirror 2718 May 15 19:37 Release
-rw-r--r-- 1 mirror mirror  650 May 15 19:37 Release.gpg
-rw-r--r-- 1 mirror mirror   32 Jul  1 21:47 Release.gpg.md5
-rw-r--r-- 1 mirror mirror   40 Jul  1 21:47 Release.gpg.sha1
-rw-r--r-- 1 mirror mirror   64 Jul  1 21:47 Release.gpg.sha256
-rw-r--r-- 1 mirror mirror   32 Jul  1 21:47 Release.md5
-rw-r--r-- 1 mirror mirror   40 Jul  1 21:47 Release.sha1
-rw-r--r-- 1 mirror mirror   64 Jul  1 21:47 Release.sha256
drwxr-xr-x 5 mirror mirror    5 Jul  1 21:47 main

Current Workaround is to delete the affected Files for Debian Repository at every Repository Synchronization.

Workaround Script:

#!/bin/bash

## Target Folder
TARGET_FOLDER="/data/salt-mirror"

# Remove existing Hashes
mapfile -t hash_files < <( find "${TARGET_FOLDER}/saltproject/deb/dists" -iwholename "*/*.sha1" -o -iwholename "*/*.sha256" -o -iwholename "*/*.md5" -o -iwholename "*/Release" -o -iwholename "*/InRelease" -o -iwholename "*/Release.gpg" )

# Loop over Hashes
echo "Begin Removal of confusing Hash Files"
for hash_file in "${hash_files[@]}"
do
    # Echo
    echo "Remove File ${hash_file}"

    # Perform Removal
    rm "${hash_file}"
done
echo "Finished Removal of confusing Hash Files"

## Mirror RPM Repository
echo "Mirror RPM Repository"
RCLONE_WEBDAV_URL=https://packages.broadcom.com/artifactory/ RCLONE_WEBDAV_VENDOR=other rclone sync --fast-list --use-server-modtime -v :webdav:saltproject-rpm/ "${TARGET_FOLDER}/saltproject/rpm/"

## Mirror DEB Repository
echo "Mirror DEB Repository"
RCLONE_WEBDAV_URL=https://packages.broadcom.com/artifactory/ RCLONE_WEBDAV_VENDOR=other rclone sync --fast-list --use-server-modtime -v :webdav:saltproject-deb/ "${TARGET_FOLDER}/saltproject/deb/"

After that on the Client Device the new Packages get properly listed and can be upgraded:

root@ubuntu:/home/user# apt policy salt-minion
salt-minion:
  Installeret: 3008.0~rc4
  Kandidat:    3008.2
  Versionstabel:
     3008.2 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3008.1 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3008.0 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
 *** 3008.0~rc4 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
        100 /var/lib/dpkg/status
     3008.0~rc3 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3008.0~rc2 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3008.0~rc1 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.14 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.13 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.12 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.11 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.10 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.9 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.8 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.7 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.6 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.5 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.4 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.3 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.2 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.1 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3007.0 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.27 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.26 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.25 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.24 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.23 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.22 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.21 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.20 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.19 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.18 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.17 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.16 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.15 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.14 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.13 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.12 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.11 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.10 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.9 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.8 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.7 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.6 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.5 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.4 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.3 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.2 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.1 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages
     3006.0 500
        500 https://salt.deb.MYDOMAIN.TLD stable/main amd64 Packages

Type of documentation
Salt Documentation.

Location or format of documentation
Air Gap Install

Additional context
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue does not name a documentation file; first locate the Air Gap Environments and rclone setup instructions, then reproduce the repository mirror update with the listed Debian metadata files. Done means the instructions reliably refresh checksums and Release metadata so apt update detects the newest package lists without manually purging /var/lib/apt/lists.

Written by the indexing model from the issue text.

Assessment

Domain
devops, documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.