PolicyEngine / PolicyEngine/policyengine-core

download_huggingface_dataset hardcodes repo_type="model", so it cannot fetch datasets repos

Open Beginner friendly
#549 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
22
Forks
30
Avg merge
14h 33m
Merged PRs (30d)
7

Description

policyengine_core.tools.hugging_face.download_huggingface_dataset (policyengine-core 3.32.5, policyengine_core/tools/hugging_face.py:92-94) always calls hf_hub_download(..., repo_type="model"). Run against the canonical US population release policyengine/populace-us (a Hugging Face datasets repo; certified tag populace-us-2024-spm-20260915), it raises RepositoryNotFoundError: 401, because there is no model repo of that name. Observed today while re-qualifying the PolicyEngine 6.0.0 bundle (registry re-qualification lane, 2026-09-15); the wrapper policyengine 6.0.0 avoids the helper by reading repo_type: "dataset" from its bundle manifest and downloading with the correct type, and recomputing the certified hash independently.

Suggested fix: accept a repo_type argument (default "model" for backward compatibility with policyengine-uk-data-private and the other model-typed data repos), or infer it by probing repo_info for both types; pass it through to hf_hub_download and to the private/gated check. This is separate from #538, which concerns token handling for public-but-gated repos.

🤖 Generated with Claude Code

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in policyengine_core/tools/hugging_face.py at download_huggingface_dataset and inspect the hf_hub_download call and private/gated check around lines 92-94. Add repository-type handling while preserving the model default, then verify that policyengine/populace-us downloads as a dataset and existing model-typed repositories still work.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python
Domain
data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.