explosion / explosion/spaCy

`spacy download --url` is silently ignored or rejected — the custom URL flag never works

Open
#13,963 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

## How to reproduce the behaviour

The `--url` flag added in #13848 to `python -m spacy download` cannot succeed under any input. Either the user-supplied URL is silently replaced with the default GitHub URL, or it is rejected by the post-construction guard.

```python
import sys
import importlib
from unittest.mock import patch

import spacy
from spacy import about

# `from spacy.cli import download` resolves to the *function*, not the module,
# because of an alias in cli/__init__.py. Force-import the module:
importlib.import_module("spacy.cli.download")
dl = sys.modules["spacy.cli.download"]

captured = {}
def fake_run(cmd):
captured["url"] = next(arg for arg in cmd if arg.startswith("http"))

# Case 1: --url WITHOUT trailing slash
captured.clear()
with patch.object(dl, "run_command", fake_run), \
patch.object(dl, "_get_pip_install_cmd", lambda: ["pip", "install"]):
dl.download_model("foo-1.0.tar.gz", custom_url="https://my-mirror.example.com/models")
print(captured["url"])
# → https://github.com/explosion/spacy-models/releases/download/foo-1.0.tar.gz
# (the user's mirror was silently discarded)

# Case 2: --url WITH trailing slash
with patch.object(dl, "run_command", fake_run), \
patch.object(dl, "_get_pip_install_cmd", lambda: ["pip", "install"]):
dl.download_model("foo-1.0.tar.gz", custom_url="https://my-mirror.example.com/models/")
# → ValueError: Download from foo-1.0.tar.gz rejected. Was it a relative path?
```

## Root cause

`spacy/cli/download.py:180-186`:

```python
base_url = custom_url if custom_url else about.__download_url__
# urljoin requires that the path ends with /, or the last path part will be dropped
if not base_url.endswith("/"):
base_url = about.__download_url__ + "/" # ← clobbers custom_url
download_url = urljoin(base_url, filename)
if not download_url.startswith(about.__download_url__):
raise ValueError(f"Download from {filename} rejected. Was it a relative path?")
```

Two interlocking defects:

1. Line 183 unconditionally replaces `base_url` with the default URL when the input lacks a trailing slash, discarding the user's choice.
2. The line-185 `startswith(about.__download_url__)` guard rejects any custom URL that is preserved, because a custom mirror by definition does not start with the GitHub URL.

Result: the `--url` flag cannot reach a non-default URL under any input.

## Impact

Users in air-gapped or mirrored environments — the exact users the feature was added for, per #13848 — believe their downloads are local but are silently being dispatched to `github.com`. Egress that operators thought they had blocked at the network layer is the only remaining safeguard.

## Your Environment

- spaCy version: 3.8.13 (also reproduces on `master` HEAD, prep for 3.8.14)
- Python version: 3.12
- Platform: macOS / Linux

PR: shaun0927/spaCy@fix/download-custom-url (incoming).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.