codeforboston / codeforboston/maple
Scrape PDFs for Bill Text
- Dominant language
- TypeScript
- Stars
- 56
- Forks
- 175
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 13
Description
**Summary**
There are 236 bills in the current session that do not have the `DocumentText` field available through the MA Legislature Document API. That does not mean there isn't DocumentText - it means that text is only available through the PDF of the bill that the legislature provides.
For these bills, we should:
* Explore the data (for bills where `content.DocumentText` is null)
* As best we can, categorize and document the cases where `DocumentText` is null
* Try to scrape the bill text from the PDF (if `DocumentText` is not available through `api.getDocument`)
* Re-run the LLM summarizer/tagger on these bills once `DocumentText` is available
There is a little exploratory work in how well we can extract text and what formats are in play, but I think our chances are good here. I suspect there are at least two types of PDFs and likely more - I've seen omnibus spending bills and Ballot Initiative bills in cursory exploration).
## Additional Resources
- Nathan pulled together a script to handle this for Ballot Initiative bills in Python using PdfPlumber - we can get likely get something similar in Typescript with a suitable library: https://github.com/nesanders/ma_ballot_bill_text_extraction
- You can query Firestore in `generalCourts/194/bills` to find all of the bills in question, but here are a few bill ids to get the exploration started: `H1`, `H18`, `H4787`, `H5008`(no longer - I manually overrode this one because it is one of the Ballot Initiative bills - but the API will still reflect the null DocumentText), and `S2539`.
Contributor guide
Assessment
This issue has not been assessed yet.