PolicyEngine / PolicyEngine/policyengine-core
Make Core dataset ingestion local-only and phase out remote artifact transfer
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 22
- Forks
- 30
- Avg merge
- 14h 33m
- Merged PRs (30d)
- 7
Description
Problem
PolicyEngine Core currently mixes two responsibilities:
- loading a dataset that is already available locally; and
- selecting, authenticating to, downloading, and sometimes uploading remote artifacts.
Simulation(dataset=...) accepts several undocumented input forms and changes their types while constructing the simulation. Dataset(require=True) can initiate a network download implicitly, while Dataset.download() supports several unrelated URI schemes. This makes construction capable of hidden network I/O and duplicates artifact-materialization behavior now owned more rigorously by policyengine.py release bundles.
Intended boundary
Core should consume local calculation inputs. It should ultimately accept only:
- a
Datasetinstance; - a
Datasetsubclass whose data already exists locally; - a local filesystem path; or
- a pandas
DataFrame.
Remote dataset selection, authentication, immutable revisions, checksum verification, caching, and provenance should be handled before Core is called. For managed PolicyEngine releases, policyengine.py should materialize the certified artifact and pass its local path to Core.
Core should not ultimately interpret hf://, gs://, release://, or HTTP URLs, nor should dataset or simulation construction initiate network activity.
Proposed migration
- Define one explicit local
DatasetInputtype and normalization function instead of changing input types withinSimulation.__init__. - Make missing local data produce a clear error rather than downloading implicitly through
Dataset(require=True). - Deprecate remote URI handling in
Simulation,Dataset.url, dataset download/upload methods, and the corresponding Core data commands. - Move downstream users of Core's transfer helpers first, including API v1's UK geography artifacts,
policyengine.py's remaining GCS geography helpers, and country-package imports. - After a documented deprecation period, remove Core's Hugging Face and GCS transfer helpers and any dependencies used only for remote artifact access.
Core must not depend on policyengine.py, because policyengine.py already depends on Core and country packages. If direct country-package remote retrieval must remain supported, the transport primitive should live in a small independent artifact package or be owned directly by those packages.
Acceptance criteria
- The accepted local dataset input forms are accurately typed and documented.
- Normalization returns one well-defined dataset representation.
- Constructing a Core
SimulationorDatasetdoes not initiate network activity. - Remote URIs fail with a specific migration error after removal.
- Managed
policyengine.pycalculations continue to materialize and verify their datasets before invoking Core. - All external imports of Core's remote-transfer helpers are migrated before those helpers are removed.
This issue intentionally defines the desired boundary but does not yet decide the release schedule or whether a separate low-level artifact package is warranted.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by mapping Simulation and Dataset construction, the Core data commands, and the remote-transfer helpers mentioned in the issue, then identify external users in API v1, policyengine.py, and country packages. Trace how local inputs and remote URIs are currently normalized or downloaded. Done means local-only inputs are typed and documented, construction performs no network activity, migration errors are specific, downstream imports are migrated, and managed calculations still materialize and verify datasets before calling Core.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- backend, data
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100