codeforboston / codeforboston/maple

Fix Bill History Scraper

Open
#2,120 0 comments 0 reactions 1 assignee Claimed by @ACoullard View on GitHub
bug scraper
Dominant language
TypeScript
Stars
56
Forks
175
Avg merge
2d 5h
Merged PRs (30d)
13

Description

## Summary

The Bill History segment of the Bills Scraper has been broken for a little while now - if a Bill History item has a `null` Date, it will break the scraper (which has a type check that does not allow BillHistoryItem.Date to be null) and fail the batch. To fix this, we should make `Date` optional (and handle that case on the frontend accordingly).

This has not impacted us severely because of the low frequency of updates and the small batch sizes of the scraper, but it also revealed a potential issue in our error reporting that this wasn't surfaced much earlier - the fact that we caught similar bugs in other firebase functions recently makes me suspect that this issue is more recent than not.

One example bill that ran into this issue is https://maple-dev.vercel.app/bills/194/H2572 - there are only three items shown in our Bill History, but there are least five items in the legislature API - one with no date, and one recent item with a Date (that failed to scrape because of the null issue)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.