GSA / GSA/data.gov

Harvest code.json files

Open
#6,318 0 comments 0 reactions 1 assignee Claimed by @jaredb96 View on GitHub
Code Repo
Dominant language
Python
Stars
1.1k
Forks
200
Avg merge
4h 56m
Merged PRs (30d)
5

Description

What: Add support for harvesting federal agency code.json files (Federal Source Code Policy format) into the Data.gov catalog. Add new schema type "code.json", transformation logic to convert code.json releases to DCAT-US 3.0 datasets, validation for both code.json structure and resulting DCAT output, and UI updates to support creating code.json harvest sources.

Why: Federal agencies are required to maintain code.json files cataloging their open-source projects. Harvesting these enables discovery of ~10,000 federal repositories through Data.gov, compliance monitoring for the Federal Source Code Policy, and unified search across both datasets and code. Example: DHS publishes 50+ repositories at `[https://www.dhs.gov/xlibrary/assets/digital-strategy/code.json`—after](https://www.dhs.gov/xlibrary/assets/digital-strategy/code.json) harvesting, each becomes searchable by name, programming language, license, and agency.

Done When:

- [x] Add "code.json" to SCHEMA_TYPE_VALUES in shared/constants.py + database migration
- [x] Create harvester/utils/codejson_mapper.py with codejson_release_to_dcat() function mapping code.json fields to DCAT-US 3.0 (repositoryURL → identifier, name → title, languages → theme, etc.)
- [x] Update harvest.py: handle code.json in __post_init__(), extract releases in acquire_minimum_external_data(), add Record.transform_codejson() method, use repositoryURL as identifier
- [x] Create harvester/utils/codejson_validator.py to validate code.json v2.0.0 structure (dual validation: code.json + transformed DCAT)
- [x] Add "code.json (Federal Source Code)" to schema type dropdown in harvest source form with help text
- [x] Write tests: unit tests for mapper (basic mapping, missing fields, email formatting), integration tests for full harvest workflow (mock code.json, dataset creation), e2e test with real DHS code.json

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.