mindependency/latestdependency: sourcing dependency versions from the Azure DevOps feed instead of PyPI — behavior change & tradeoffs

## Summary

The `mindependency` (and sibling `latestdependency`) checks currently resolve dependency versions by querying the **PyPI JSON API** (`https://pypi.org/pypi//json` and `...///json`). In network-isolated CI configurations PyPI is unreachable, which causes these checks to fail during dependency resolution — **before any recorded tests run** (e.g. [buildId=6617297](https://dev.azure.com/azure-sdk/public/_build/results?buildId=6617297)).

An alternative backend already exists in the branch under exploration: `pypi_tools/azdo.py` (`AzureArtifactsClient`) resolves versions from the **Azure DevOps Artifacts feed** (`azure-sdk-for-python`) instead of PyPI. It is auto-selected when `PIP_INDEX_URL` points at `pkgs.dev.azure.com/.../pypi/simple/`.

This issue documents **what changes**, **the tradeoffs of sourcing version metadata from the feed instead of PyPI**, and the **investigation evidence** behind those tradeoffs.

## What changes

- **Version enumeration** moves from the PyPI JSON API to the AzDO Artifacts **Feed REST API** (`feeds.dev.azure.com/.../_apis/packaging/Feeds/{id}/packages?protocolType=pypi&includeAllVersions=true`).
- **`requires_python`** (needed to filter Python-incompatible versions) is *not* in the REST API and would come from the PEP 503 simple index (`data-requires-python`).
- **`requires_dist`** (needed by the dev-requirement compatibility walk) is *not* in any feed API/index — it exists only inside each artifact's `METADATA`/`PKG-INFO`, so it requires downloading the wheel/sdist.
- Net effect for a typical package: version listing becomes **feed-native and faster**, but a few correctness/coverage gaps appear (below).

## Tradeoffs / caveats

### Correctness-affecting
1. **Feed lists only *cached* versions, not all of PyPI.** The feed is a proxy cache, not a mirror. A version nobody has pulled is invisible until requested by exact version (which triggers an on-demand upstream pull). Old minimums are the most likely to be missing.
2. **"Latest" through the feed = latest *cached*, not latest *released*.** Verified: pip/feed sees `azure-core` latest `1.40.0` while PyPI has `1.41.0`. `latestdependency` can therefore silently test a **stale** latest.
3. **`requires_python` filtering is silently dropped** on the AzDO backend (`filter_by_compatibility is not supported against Azure Artifacts; returning unfiltered versions`) — a minimum incompatible with the running Python could be selected.
4. **Yanked status is not propagated.** The feed served `requests` `2.29.0/2.32.0/2.32.1/2.32.2`, which PyPI hides as yanked. Only hard `isDeleted` exists — a yanked version could be installed.

### Coverage / fidelity
5. **Azure packages are safe; third-party deps are the risk.** `azure-core` had full history (1748 versions) because CI publishes every build; `isodate`/`typing-extensions` are sparsely proxy-cached.
6. **No PyPI JSON API on the feed** (`/pypi//json` → 404). Only the simple index + AzDO REST API.
7. **No `requires_python`/`requires_dist` in REST/JSON objects** (only `version`, `publishDate`, `isDeleted`, `isLatest`).
8. **No PEP 658** — the `.metadata` sidecar 404s, so `requires_dist` means downloading the whole artifact (~190 KB/wheel).
9. **`publishDate` ≠ PyPI upload date** (it's the AzDO cache date) — date-based filtering is approximate.
10. **Only the first-requested platform's wheel is cached** (per MS docs) — cross-platform installs of the same version can fail until each wheel is pulled.

### Auth / scope / perf
11. **The REST API and raw simple index require auth even on the *public* feed** (anonymous simple-index HTTP 302-redirects to sign-in). CI uses `PipAuthenticate@1` to embed creds in `PIP_INDEX_URL`; `parse_pip_index_url` extracts the embedded PAT.
12. **`project()`/`filter_packages_for_compatibility()` raise `NotImplementedError` on the AzDO backend, and `project_release()` hard-falls-back to `pypi.org`** — so the dev-req `requires_dist` walk reintroduces PyPI calls even "on the feed."
13. **Other `PyPIClient` consumers still assume pypi.org** (`ci_tools/functions.py`, `ci_tools/dependency_analysis.py`, `packaging_tools/code_report.py`, `scripts/discover_unpublished_packages.py`) — making `mindependency` feed-only does not make the whole tool PyPI-free.
14. **Performance is mixed:** enumeration is much faster (6 feed requests / ~1.9s vs 152 PyPI requests / ~7–19s for `azure-keyvault-secrets`), but `requires_dist` gets slower (full artifact download per candidate; needs caching + early-exit).

### Suggested follow-up work
- Implement `requires_python` on `AzureArtifactsClient` (from the simple index `data-requires-python`) and wire it into `get_ordered_versions(..., filter_by_compatibility=True)` so the filter is no longer dropped.
- Source `requires_dist` from artifact `METADATA` (or gate the pypi.org fallback behind a flag for isolated networks).
- Decide policy for yanked versions (accept risk vs. static deny-list).
- Ensure the feed is pre-warmed for the minimum/latest third-party versions the checks depend on, or pin-and-pull exact versions.
- Audit the other `PyPIClient` call sites so they don't silently re-hit pypi.org.

Investigation details & raw outputs (click to expand)

All probes were run against the public `azure-sdk/public` `azure-sdk-for-python` feed. Read-only probes did **not** mutate the feed cache (no artifact file was downloaded). Authenticated probes used an AAD token from `az account get-access-token --resource 499b84ac-1321-427f-aa17-267ca6975798`.

### 1. Where the tool calls PyPI

Primary call site — `ci_tools/scenario/dependency_resolution.py` `process_requirement()`:

```python
client = PyPIClient()
versions = [str(v) for v in client.get_ordered_versions(pkg_name, True)] # filter_by_compatibility=True
```

On the PyPI backend this is `1` `GET /pypi/{pkg}/json` (version list) + `N` `GET /pypi/{pkg}/{version}/json` (per-version `requires_python`). Second call site: `ci_tools/functions.py` `resolve_compatible_package()` (`requires_dist` walk).

### 2. Live `mindependency` resolution for `azure-keyvault-secrets` (PyPI backend)

Instrumented `find_released_packages(pkg_dir, "Minimum")` and logged every HTTP request.

```
=== RESOLVED minimum versions to install ===
isodate==0.6.1
azure-core==1.31.0
typing-extensions==4.6.0

=== PyPI request summary ===
total HTTP requests : 152
version-list calls : 3
per-version calls : 149 # one GET /pypi/{pkg}/{ver}/json per version, for requires_python

=== per-dependency breakdown ===
azure-core list=1 per-version=80 total=81
isodate list=1 per-version=17 total=18
typing-extensions list=1 per-version=52 total=53
```

### 3. Same resolution against the authenticated AzDO feed backend

`PIP_INDEX_URL=https://user:@pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-python/pypi/simple/`

```
backend: azdo
WARNING:root:filter_by_compatibility is not supported against Azure Artifacts; returning unfiltered versions (x3)

=== RESULT ===
isodate==0.6.1
azure-core==1.31.0
typing-extensions==4.6.0 # identical minimums

=== HTTP request summary ===
total HTTP requests : 6
feeds.dev.azure.com: 6 # 3x resolve-feed-id + 3x packages list
pypi.org hits : 0
elapsed : 1.9s

# feed version counts returned per dependency:
isodate feed versions: 4 (0.6.0 .. 0.7.2)
azure-core feed versions: 1748 (1.1.0.dev... .. 1.41.0a...)
typing-extensions feed versions: 16 (4.2.0 .. 4.16.0)
```

### 4. Feed enumeration is a strict subset of PyPI (read-only diff)

```
=== requests === PyPI: 163 Feed: 14 in-PyPI-not-feed: 149 in-feed-not-PyPI: 0
=== six === PyPI: 29 Feed: 3 in-PyPI-not-feed: 26 in-feed-not-PyPI: 0
=== flask === PyPI: 64 Feed: 6 in-PyPI-not-feed: 58 in-feed-not-PyPI: 0
```

### 5. "Latest" via pip through the feed vs PyPI (`pip index versions`, read-only)

```
===== azure-core =====
via DevOps feed : Available versions: 1.40.0, 1.39.0, 1.38.3, ... (max = 1.40.0)
via PyPI : Available versions: 1.41.0, 1.40.0, 1.39.0, ... (max = 1.41.0)

===== requests =====
via DevOps feed : ... 2.34.2 ... plus 2.32.2/2.32.1/2.32.0/2.29.0 (yanked on PyPI, still served by feed)
via PyPI : ... 2.34.2 ... (yanked versions hidden)
```

Latest-vs-latest table:

| Package | PyPI latest | Feed latest |
|---|---|---|
| requests | 2.34.2 | 2.34.2 (already cached) |
| six | 1.17.0 | 1.17.0 |
| typing-extensions | 4.16.0 | 4.16.0 |
| isodate | 0.7.2 | 0.7.2 |
| **azure-core** | **1.41.0** | **1.40.0 (lags)** |

### 6. Endpoint metadata availability (empirically probed)

| Data point | PyPI JSON | AzDO Feed REST | AzDO Python-specific | AzDO simple index |
|---|---|---|---|---|
| Version list | ✅ | ✅ `includeAllVersions=true` | ❌ single version only | ⚠️ stale/cached only |
| requires_python | ✅ | ❌ | ❌ | ⚠️ `data-requires-python` |
| requires_dist | ✅ | ❌ | ❌ | ❌ |
| Upload date | ✅ per file | ⚠️ `publishDate` = cache date | ❌ | ❌ |
| Yanked | ✅ | ❌ (only `isDeleted`) | ❌ | ⚠️ `data-yanked` (0 hits observed) |
| PyPI JSON shape | ✅ | ❌ 404 | ❌ | ❌ |
| PEP 658 `.metadata` | n/a | n/a | n/a | ❌ (404) |

`requires_python` + `requires_dist` are both present in the wheel's `METADATA` (and sdist `PKG-INFO`):

```
azure_core-1.30.0.dist-info/METADATA
Requires-Python: >=3.7
Requires-Dist: requests >=2.21.0
Requires-Dist: six >=1.11.0
Requires-Dist: typing-extensions >=4.6.0
Requires-Dist: aiohttp >=3.0 ; extra == 'aio'
```

### 7. Strict `>` dependency specifiers

Repo-wide scan of `pyproject.toml`/`setup.py`: **257** `>=` specifiers, **0** strict `>` specifiers. So the "next version after X" fragility does not affect direct deps today, but would if a `>` bound were introduced (or via transitive deps at pip install time).

Start with ci_tools/scenario/dependency_resolution.py and pypi_tools/azdo.py, then inspect ci_tools/functions.py for the requires_dist walk. Reproduce dependency resolution for azure-keyvault-secrets against PyPI and the authenticated Azure DevOps feed, and compare the documented metadata, compatibility, freshness, and fallback behavior.

Written by the indexing model from the issue text.