apache / apache/arrow

[Packaging][Release] Binary verification downloads loads of packages which aren't necessary for the verification

Open
#49,242 0 comments 0 reactions 0 assignees View on GitHub
Component: Packaging Component: Release Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Verification of binaries when executing the following:
```
TEST_DEFAULT=0 TEST_BINARY=1 ./dev/release/verify-release-candidate.sh 23.0.1 0
```

Takes a really long time.
I've noticed we seem to be downloading a lot of binaries that don't seem related to the actual release being verified, a really small snippet example:
```
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_12-1_amd64.deb to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_12-1_amd64.deb
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_12-1_amd64.deb.asc to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_12-1_amd64.deb.asc
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_12-1_amd64.deb.sha512 to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_12-1_amd64.deb.sha512
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_13-1_amd64.deb to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_13-1_amd64.deb
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_13-1_amd64.deb.asc to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_13-1_amd64.deb.asc
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_13-1_amd64.deb.sha512 to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_13-1_amd64.deb.sha512
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_14-1_amd64.deb to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_14-1_amd64.deb
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_14-1_amd64.deb.asc to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_14-1_amd64.deb.asc
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_14-1_amd64.deb.sha512 to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_14-1_amd64.deb.sha512
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_15-1_amd64.deb to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_15-1_amd64.deb
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_15-1_amd64.deb.asc to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_15-1_amd64.deb.asc
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_15-1_amd64.deb.sha512 to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_15-1_amd64.deb.sha512
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_16-1_amd64.deb to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_16-1_amd64.deb
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_16-1_amd64.deb.asc to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_16-1_amd64.deb.asc
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_16-1_amd64.deb.sha512 to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_16-1_amd64.deb.sha512
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_17-1_amd64.deb to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_17-1_amd64.deb
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_17-1_amd64.deb.asc to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_17-1_amd64.deb.asc
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_17-1_amd64.deb.sha512 to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_17-1_amd64.deb.sha512
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_18-1_amd64.deb to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_18-1_amd64.deb
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_18-1_amd64.deb.asc to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_18-1_amd64.deb.asc
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_18-1_amd64.deb.sha512 to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_18-1_amd64.deb.sha512
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_19-1_amd64.deb to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_19-1_amd64.deb
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_19-1_amd64.deb.asc to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_19-1_amd64.deb.asc
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_19-1_amd64.deb.sha512 to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_19-1_amd64.deb.sha512
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_20-1_amd64.deb to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_20-1_amd64.deb
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_20-1_amd64.deb.asc to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_20-1_amd64.deb.asc
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_20-1_amd64.deb.sha512 to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_20-1_amd64.deb.sha512
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_21-1_amd64.deb to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_21-1_amd64.deb
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_21-1_amd64.deb.asc to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_21-1_amd64.deb.asc
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_21-1_amd64.deb.sha512 to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_21-1_amd64.deb.sha512
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_22-1_amd64.deb to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_22-1_amd64.deb
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_22-1_amd64.deb.asc to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_22-1_amd64.deb.asc
Downloading debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_22-1_amd64.deb.sha512 to /tmp/arrow-23.0.1.EbVUu/binaries/debian-rc/pool/bookworm/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_22-1_amd64.deb.sha512

```
In this case those are adbc and we download several versions of them which are unnecessary.

The problem gets worst when for connectivity issues the verification fails and we have to restart again.

I'll try to time it to get a real number of how long it takes but I am sure it takes ~2 hours.

### Component(s)

Release, Packaging

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.