CodeForPhilly / CodeForPhilly/clean-and-green-philly

Task: Data validation subticket #3: External input validators

Open
#1,224 3 comments 0 reactions 1 assignee Claimed by @gabecano4308 View on GitHub
high priority
Dominant language
Python
Stars
50
Forks
92
PR merge metrics
No merged PRs in 30d

Description

## Subticket 3: External Input Validators

### Describe the task
Frequently, we have had issues with APIs or external sources changing without us realizing and then messing up the pipeline. Create validators that define our expectations for the structure of external sources so that we can identify what has changed and when to prevent bad data from getting into production and make troubleshooting easier.

### Acceptance Criteria
- [ ] Identify all API/external input sources requiring validation
- [ ] Implement validator for each API/input confirming relevant columns still exist and shape of data is within an acceptable margin of difference
- [ ] Implement validator for each API/input confirming columns have expected data
- [ ] Add test coverage for all API validators
- [ ] Document any discrepancies found vs. expected schema

### Additional context
- Nissim can provide expected column names and data types for each API
- Flag any unexpected schema changes as separate tickets

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.