CouncilDataProject / CouncilDataProject/cdp-scrapers

Include diff file in test-scraper.yml Action artifact

Open
#93 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
27
Forks
18
PR merge metrics
No merged PRs in 30d

Description

### Feature Description

A comment in the form `/test-scraper --from_dt= --to_dt= get_{municipality_slug}_events` in a pull request or an issue triggers the GitHub Action `.github/workflows/test-scraper.yml`. The Action uploads a ZIP file as an artifact. The ZIP contains a JSON file for the scraped events returned by `cdp_scrapers.instances.get_{municipality_slug}_events()`.

If the Action was triggered by a pull request comment, it would be helpful to also include a diff file, comparing the results from the same call between the pull target branch (`CouncilDataProject/cdp-scrapers main`) and the pull merge branch.

Change summary:
1. Do something similar to what `test-scraper.yml` does currently: save scraped events in JSON file.
2. Repeat, using `CouncilDataProject/cdp-scrapers main`.
3. Save output from diff between the two JSON files.
4. Add the diff file in the ZIP file artifact for the Action.

### Use Case

Often we find ourselves manually creating this kind of diff files to study changes (if any) some new code would yield in the scraped `IngestionModels`. Doing so automatically for a pull request `/test-scraper` run would reduce manual effort.

### Solution

File(s) in `.github/workflows/` (and subdirectories) updated as necessary to include diff files, described above, in the artifact ZIP.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.