Scope and document CSW harvest source type
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 200
- Avg merge
- 4h 56m
- Merged PRs (30d)
- 5
Description
**What:** Some federal agencies publish ISO 19115 records via CSW (Catalog Service for the Web) endpoints rather than WAFs. CSW is an OGC-standard query protocol that supports filtered harvest by date, extent, and keyword. The harvester currently has no CSW source type.
This ticket scopes the CSW source type -- it does not build it. The output is a technical design document or GitHub epic that can be estimated and scheduled.
**Scope the following:**
- Identify which agencies currently have active CSW endpoints that would benefit from a Data.gov harvest path (NOAA NCEI, USGS, EPA, NGA are likely candidates)
- Document the CSW GetCapabilities / GetRecords / GetRecordById interaction pattern
- Define the CSW source type schema: endpoint URL, filter parameters, output format, max records per page, authentication requirements
- Map CSW pagination to the WAF-like `modified_date` filtering the harvester already uses
- Identify whether existing Python CSW client libraries (e.g., OWSLib) are suitable or whether custom logic is needed
- Estimate implementation effort
**Done When:**
- A scoping document is produced and attached to this ticket
- Known CSW endpoints and agencies are listed
- Effort estimate is produced
- A follow-on implementation epic is written if the work is approved
Contributor guide
Assessment
This issue has not been assessed yet.