GSA / GSA/data.gov

Scope and document CSW harvest source type

Open
#6,045 1 comment 0 reactions 1 assignee Claimed by @rshewitt View on GitHub
Harvester
Dominant language
Python
Stars
1.1k
Forks
200
Avg merge
4h 56m
Merged PRs (30d)
5

Description

**What:** Some federal agencies publish ISO 19115 records via CSW (Catalog Service for the Web) endpoints rather than WAFs. CSW is an OGC-standard query protocol that supports filtered harvest by date, extent, and keyword. The harvester currently has no CSW source type.

This ticket scopes the CSW source type -- it does not build it. The output is a technical design document or GitHub epic that can be estimated and scheduled.

**Scope the following:**
- Identify which agencies currently have active CSW endpoints that would benefit from a Data.gov harvest path (NOAA NCEI, USGS, EPA, NGA are likely candidates)
- Document the CSW GetCapabilities / GetRecords / GetRecordById interaction pattern
- Define the CSW source type schema: endpoint URL, filter parameters, output format, max records per page, authentication requirements
- Map CSW pagination to the WAF-like `modified_date` filtering the harvester already uses
- Identify whether existing Python CSW client libraries (e.g., OWSLib) are suitable or whether custom logic is needed
- Estimate implementation effort

**Done When:**
- A scoping document is produced and attached to this ticket
- Known CSW endpoints and agencies are listed
- Effort estimate is produced
- A follow-on implementation epic is written if the work is approved

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.