[FEA] Support a mirror base URL for Arrow bundled Thrift downloads
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
When cuDF enables Arrow Parquet, `cpp/cmake/thirdparty/get_arrow.cmake` selects `Thrift_SOURCE BUNDLED`. Arrow then downloads the Thrift source archive during its nested CMake configure. That download currently depends on Arrow’s default upstream URLs unless every caller constructs and supplies the complete versioned `ARROW_THRIFT_URL` itself.
This makes cuDF builds unnecessarily vulnerable to upstream availability and prevents CI systems from configuring a stable authenticated mirror using only a mirror base URL. [rapidsai/cudf#23409](https://github.com/rapidsai/cudf/pull/23409) is an example of an affected PR: its build was blocked while Arrow’s bundled Thrift archive could not be fetched reliably.
**Describe the solution you would like**
Add generic mirror plumbing alongside cuDF’s Arrow dependency setup:
- Accept an environment variable such as `ARROW_THRIFT_MIRROR_URL` containing the mirror base URL.
- During the Arrow project configure, read `ARROW_THRIFT_BUILD_VERSION` from Arrow’s `thirdparty/versions.txt` and derive the versioned archive URL, for example `${ARROW_THRIFT_MIRROR_URL}/thrift//thrift-.tar.gz`.
- Set Arrow’s supported `ARROW_THRIFT_URL` override only when the caller has not already supplied `ARROW_THRIFT_URL`, so the explicit full URL remains authoritative.
- Leave existing behavior unchanged when no mirror variable is defined.
A CMake project hook (for example, a helper installed through `CMAKE_PROJECT_arrow_INCLUDE` from `get_arrow.cmake`) can run after Arrow establishes its source directory and before it resolves bundled dependencies.
Authentication remains the responsibility of each build environment: callers can configure `CMAKE_NETRC` and `CMAKE_NETRC_FILE` as appropriate. cuDF would own only the reusable Arrow/Thrift URL construction.
**Describe alternatives you have considered**
- Have every downstream repository parse Arrow’s version file and install its own project hook. This duplicates Arrow-specific dependency logic and can drift across cuDF consumers.
- Hard-code the complete Thrift URL in CI. This duplicates the Thrift version and must be updated whenever Arrow changes it.
- Rely only on Arrow’s public fallback URLs. This does not address transient upstream failures or authenticated enterprise mirrors.
**Additional context**
The Java JNI build enables Arrow Parquet through cuDF’s `get_arrow.cmake`, so placing the generic mechanism there would cover cuDF itself and downstream builds that configure `java/src/main/native`.
Contributor guide
Research direction
Start in cpp/cmake/thirdparty/get_arrow.cmake and trace the nested Arrow configure, including how Arrow reads thirdparty/versions.txt and resolves bundled Thrift. The work is complete when a mirror base URL derives the versioned Thrift archive, an explicitly supplied ARROW_THRIFT_URL remains authoritative, and builds without the mirror retain existing behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cmake, cpp
- Domain
- build-system
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100