pytorch / pytorch/pytorch

Audit license-files glob in pyproject.toml for over-collection

Open
#183,434 2 comments 0 reactions 0 assignees View on GitHub
bot-triaged enhancement module: binaries module: third_party needs design triaged
Dominant language
Python
Stars
103k
Forks
29.5k
PR merge metrics
PR metrics pending

Description

### 🚀 The feature, motivation and pitch

#180237 migrates PyTorch's bundled-license shipping mechanism from the legacy `setup.py`-concatenated `LICENSE` to the PEP 639 `license-files` field. As part of that PR's review, the question came up of whether the (old and new) `license-files` glob accurately represents the distribution's licensing surface.

The PR uses a minimal recursive glob that is designed to be identical to the old collection logic:

```toml
license-files = [
"LICENSE",
"third_party/**/LICENSE",
"third_party/**/LICENSE.txt",
"third_party/**/LICENSE.rst",
"third_party/**/COPYING.BSD",
]
```

This matches ~108 files per built wheel. Inspection shows it over-collects in several categories:

| Category | Examples |
|---|---|
| **Test fixtures (most concerning -- one is GPL-3.0)** | `kineto/libkineto/third_party/dynolog/third_party/cpr/test/LICENSE` (GPL-3.0), `fbgemm/fbgemm_gpu/test/quantize/mx/LICENSE` |
| **Test frameworks** | `googletest/LICENSE` and 10+ copies of `googletest/`, `googlemock/`, `gtest/`, `doctest/` across nested submodules |
| **Documentation pages** (not license text) | `composable_kernel/docs/LICENSE.rst` and copies in aiter, fbgemm, flash-attention, mslk; `fbgemm/fbgemm_gpu/docs/src/general/LICENSE.rst`; `NVTX/tools/docs/github-markdown-css/LICENSE` |
| **Build/lint tools** (Python sources, not in any binary) | `hipify_torch/LICENSE.txt` (3 copies), `kineto/.../json/third_party/cpplint/LICENSE` |
| **Language bindings PyTorch doesn't ship** | `flatbuffers/dart/LICENSE`, `flatbuffers/swift/LICENSE`, `cutlass/python/LICENSE.txt` (4 copies), `NVTX/python/LICENSE.txt`, `nccl/bindings/nccl4py/LICENSE.txt` |
| **Example / sample code** | `prometheus-cpp/3rdparty/civetweb/examples/rest/cJSON/LICENSE`, two `duktape-*/LICENSE.txt` copies under civetweb |
| **dynolog's transitive deps** (never compiled) | `dynolog/third_party/{cpr,DCGM,fmt,pfs,prometheus-cpp}/LICENSE*` and their nested transitive deps |
| **Redundant duplicates** | `NVTX/docs/LICENSE.txt` and `NVTX/python/LICENSE.txt` are byte-identical copies of `NVTX/LICENSE.txt` |

After conservative exclusion, ~61 of the 108 files represent code that actually ships in the wheel; the remainder are over-collection artifacts of the recursive glob.

This is pre-existing behaviour. The old `setup.py` `concat_license_files` mechanism that #180237 replaces walked `third_party/` with the same broad criteria and produced an equivalently over-collected `LICENSE` blob in every release wheel. The PEP 639 layout just makes it visible per-file rather than buried in concatenated text.

The technical layout migration (#180237) is in scope to land first; this issue is the follow-up to discuss and implement the audit of *which* files belong in the set.

cc @malfet @atalman @tinglvv @nWEIdia @rgommers @seemethere

### Alternatives

In rough order of investment:

1. **Tighten the recursive globs to a finite depth.** Five explicit-depth patterns (`third_party/*/LICENSE*` through `third_party/*/*/*/*/*/LICENSE*`) drop all files at path depth >= 8, which removes the GPL-3.0 cpr/test/LICENSE and most dynolog-transitive over-collection. Doesn't address shallower over-collection (googletest at depth 3, cutlass/python at depth 4, etc.).

2. **Enumerate `license-files` explicitly per shipped submodule.** Matches NumPy's pattern. ~61 paths, sorted, each line auditable. Most precise; adds maintenance burden whenever a vendored submodule moves or a new one is added.

3. **Compute `license-files` dynamically.** Use `dynamic = ["license-files"]` with a build-backend hook that walks `third_party/` and applies exclude patterns. Setuptools 77+ may support this; scikit-build-core may need a custom metadata provider. Most flexible, biggest infra investment.

4. **Pair the chosen approach with the SPDX `license` expression update.** @rgommers' review on #180237 also asked for the SPDX `license` field to reflect the actual distribution licenses; that's coupled to whichever subset of files we decide ships, so should be tackled in the same pass.

### Additional context

#### Why cpr/test deserves a specific call-out

`third_party/kineto/libkineto/third_party/dynolog/third_party/cpr/test/LICENSE` is GPL-3.0. Including it in the distribution's `license-files` set would, on a strict reading of PEP 639, imply the distribution contains GPL-3.0 code.

It does not. The full chain of gates:

1. PyTorch only invokes `libkineto`'s CMakeLists, not `dynolog`'s top-level CMakeLists. So `dynolog_lib` (the full dynolog with cpr) is never built.
2. libkineto only links the `dynolog_ipcfabric_lib` target, which is `add_library(... INTERFACE)` -- an empty INTERFACE library with no compiled sources.
3. Even if `dynolog_lib` were built, `add_subdirectory(third_party/cpr)` is gated by `if(USE_ODS_GRAPH_API)`, a Facebook-internal flag that PyTorch never enables.
4. Even if cpr were built, `CPR_BUILD_TESTS` defaults to OFF, so `cpr/test/CMakeLists.txt` is never added.

So no GPL-3.0 code from cpr/test is ever compiled, linked, or executed in any PyTorch build -- but the LICENSE file is structurally present in the source tree because of recursive submodule vendoring, and the recursive `license-files` glob picks it up.

#### Scope of this issue

Discussion of which approach to take and on which timeline. Implementation is a separate PR (or set of PRs), ideally after #180237 has landed so the PEP 639 layout is the stable baseline.

#### Refs

- #180237 (the PEP 639 layout migration)
- #158104 (earlier PEP 639 attempt, reverted twice for an unrelated torchvision-on-macOS issue)
- rgommers' inline review on #180237 (`pyproject.toml:55`) re: SPDX expression accuracy

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.