download_huggingface_dataset sends no token for public+gated repos (private is the wrong predicate)

Summary

download_huggingface_dataset decides whether to authenticate by testing ModelInfo.private. For a repo that is public but gated, private is False, so no token is sent and the gate returns 401 GatedRepoError — even when a valid, gate-approved token is available in the environment.

policyengine_core/tools/hugging_face.py:81-90:

fetched_model_info: ModelInfo = model_info(repo)
is_repo_private: bool = fetched_model_info.private

authentication_token: str = None
if is_repo_private:
    authentication_token: str = get_or_prompt_hf_token()

return hf_hub_download(
    repo_id=repo,
    ...
    token=authentication_token,   # None for public+gated
)

Reproduction

policyengine/policyengine-uk-data-private was flipped from private to public + gated (manual approval) on 31 July 2026:

>>> from huggingface_hub import model_info
>>> i = model_info("policyengine/policyengine-uk-data-private")
>>> i.private, i.gated
(False, 'manual')

Any download_huggingface_dataset call against it now fails:

huggingface_hub.errors.GatedRepoError: 401 Client Error.
Cannot access gated repo for url https://huggingface.co/policyengine/policyengine-uk-data-private/resolve/1.40.3/enhanced_frs_2023_24.h5.
Access to model policyengine/policyengine-uk-data-private is restricted. You must have access to it and be authenticated to access it.

Impact

This took down every dataset-backed CI job in policyengine-uk from 31 July to 11 August (PolicyEngine/policyengine-uk#1816). The failure mode is unusually expensive to diagnose because it is indistinguishable from a bad credential: the error says "you must be authenticated", so the natural response is to rotate the token. That was done on 10 August and changed nothing, because no token was being sent. Validating the replacement token out of band also passed — that check supplied the token explicitly, which is the step this function skips.

Any country package that downloads from a gated HF repo has the same latent bug, and gated-instead-of-private is the posture HF actually recommends for licensed data, since access grants are no-ops on private repos.

Suggested fix

The private test made sense when it was added (#320) — the only restricted repos were private ones. It no longer partitions the space. Two options:

1. Pass the token whenever one is available. Simplest and hard to get wrong; hf_hub_download ignores a token it does not need for a genuinely public repo.
2. Test gated alongside private: if fetched_model_info.private or fetched_model_info.gated:. Note gated is False | 'auto' | 'manual', so truthiness works but the tri-state deserves a comment.

Worth noting that dropping the explicit token entirely would also work, since huggingface_hub falls back to the HF_TOKEN environment variable when token=None (get_token_to_send → get_token) — but only for HF_TOKEN, not PolicyEngine's HUGGING_FACE_TOKEN, so the fallback is silent and name-dependent. Better to pass it explicitly.

A regression test would need a gated fixture repo, or a mock asserting the token reaches hf_hub_download when model_info reports private=False, gated='manual'.

Workaround in the meantime

Export HF_TOKEN alongside HUGGING_FACE_TOKEN so huggingface_hub picks it up implicitly: PolicyEngine/policyengine-uk#1817.

Start in policyengine_core/tools/hugging_face.py:81-90 and trace get_or_prompt_hf_token before reviewing how hf_hub_download receives its token. Add regression coverage for model_info reporting private=False and gated='manual', and confirm the download sends an available token without breaking genuinely public repositories.