NVIDIA / NVIDIA/cuvs

[BUG] Downloading and decompressing cuvs_conda_cpp_libcuvs_x86_64_cu12 occasionally takes a very long time

Open
#2,500 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Build
Dominant language
Cuda
Stars
854
Forks
236
Avg merge
3d 3h
Merged PRs (30d)
62

Description

Describe the bug
I keep noticing cuvs-java and cuvs-lucene builds that occasionally take a very long time to complete. It looks like the majority of the time is spent on these two lines (depending on the type of the build):

https://github.com/NVIDIA/cuvs/blob/fdd404f063639afb5738ad13d6868d03f50f1fe4/ci/build_java.sh#L29

https://github.com/NVIDIA/cuvs/blob/fdd404f063639afb5738ad13d6868d03f50f1fe4/ci/build_lucene.sh#L34

Most of the time these lines take just a couple of minutes. For example, in https://github.com/NVIDIA/cuvs/actions/runs/32545903651/job/96968831522?pr=2499#step:13:72 this line is executed pretty fast:

RAPIDS logger » [08/22/26 02:58:18]
┌────────────────────────────────────────────────┐
|    Downloading artifacts from previous jobs    |
└────────────────────────────────────────────────┘

[rapids-download-from-github] Detected CI=true. Skipping running 'gh auth status'.
[rapids-github-run-id] Detected CI=true. Skipping running 'gh auth status'.
[rapids-github-run-id] Assuming artifacts should be fetched from this same run of 'pr.yaml'
Downloading and decompressing cuvs_conda_cpp_libcuvs_x86_64_cu13 from Run ID 32545903651 into /tmp/tmp.m7yJdKDcm7

RAPIDS logger » [08/22/26 03:00:40]

However, in other cases such as https://github.com/NVIDIA/cuvs/actions/runs/32545903651/job/96968831551?pr=2499#step:13:72 it can take almost 30 minutes

RAPIDS logger » [08/22/26 02:59:48]
┌────────────────────────────────────────────────┐
|    Downloading artifacts from previous jobs    |
└────────────────────────────────────────────────┘

[rapids-download-from-github] Detected CI=true. Skipping running 'gh auth status'.
[rapids-github-run-id] Detected CI=true. Skipping running 'gh auth status'.
[rapids-github-run-id] Assuming artifacts should be fetched from this same run of 'pr.yaml'
Downloading and decompressing cuvs_conda_cpp_libcuvs_x86_64_cu12 from Run ID 32545903651 into /tmp/tmp.nC3Cq1kuVk

RAPIDS logger » [08/22/26 03:28:29]

And occasionally it can take almost hour and a half https://github.com/NVIDIA/cuvs/actions/runs/32545903651/job/96973240844?pr=2499

RAPIDS logger » [08/22/26 03:36:51]
┌────────────────────────────────────────────────┐
|    Downloading artifacts from previous jobs    |
└────────────────────────────────────────────────┘

[rapids-download-from-github] Detected CI=true. Skipping running 'gh auth status'.
[rapids-github-run-id] Detected CI=true. Skipping running 'gh auth status'.
[rapids-github-run-id] Assuming artifacts should be fetched from this same run of 'pr.yaml'
Downloading and decompressing cuvs_conda_cpp_libcuvs_x86_64_cu13 from Run ID 32545903651 into /tmp/tmp.0HAanNDBql
[rapids-download-from-github] Detected CI=true. Skipping running 'gh auth status'.
[rapids-github-run-id] Detected CI=true. Skipping running 'gh auth status'.
[rapids-github-run-id] Assuming artifacts should be fetched from this same run of 'pr.yaml'
Downloading and decompressing cuvs-java-cuda13.3.0 from Run ID 32545903651 into /tmp/tmp.bgbx0u4QWB

RAPIDS logger » [08/22/26 05:02:21]

I actually watched this one and it was stuck on cuvs_conda_cpp_libcuvs_x86_64_cu13 for most of that time.

Steps/Code to reproduce bug
Start a build in CI that builds cuvs-java and cuvs-lucene.

Expected behavior
Download is performed in reasonable and predictable time.

Environment details (please complete the following information):
CI

Additional context
N/A

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ci/build_java.sh line 29 and ci/build_lucene.sh line 34, then reproduce a CI build for cuvs-java and cuvs-lucene. Compare the artifact download logs for the cuvs_conda_cpp_libcuvs_x86_64_cu12 and cu13 artifacts across runs. Done means these downloads complete in a reasonable and predictable time.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, shell
Domain
build-system, ci-cd
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.