CouncilDataProject / CouncilDataProject/cdp-scrapers
Include diff file in test-scraper.yml Action artifact
- Dominant language
- Python
- Stars
- 27
- Forks
- 18
- PR merge metrics
- No merged PRs in 30d
Description
### Feature Description
A comment in the form `/test-scraper --from_dt= --to_dt= get_{municipality_slug}_events` in a pull request or an issue triggers the GitHub Action `.github/workflows/test-scraper.yml`. The Action uploads a ZIP file as an artifact. The ZIP contains a JSON file for the scraped events returned by `cdp_scrapers.instances.get_{municipality_slug}_events()`.
If the Action was triggered by a pull request comment, it would be helpful to also include a diff file, comparing the results from the same call between the pull target branch (`CouncilDataProject/cdp-scrapers main`) and the pull merge branch.
Change summary:
1. Do something similar to what `test-scraper.yml` does currently: save scraped events in JSON file.
2. Repeat, using `CouncilDataProject/cdp-scrapers main`.
3. Save output from diff between the two JSON files.
4. Add the diff file in the ZIP file artifact for the Action.
### Use Case
Often we find ourselves manually creating this kind of diff files to study changes (if any) some new code would yield in the scraped `IngestionModels`. Doing so automatically for a pull request `/test-scraper` run would reduce manual effort.
### Solution
File(s) in `.github/workflows/` (and subdirectories) updated as necessary to include diff files, described above, in the artifact ZIP.
Contributor guide
Assessment
This issue has not been assessed yet.