hasadna / hasadna/okscraper

create class/es that allows to scrape documents using open office

Open
#1 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Today in Open Knesset the scraping of documents is done differently for each scraping task

for example -
rtf documents are scraped using Rtf15Reader
doc documents are scraped using antiword - which converts the doc to xml which is then parsed

I want all the scrapers to work in a similar way, a possible solution is using Libre Office

see here for a description of using Libre Office to convert documents:
http://ask.libreoffice.org/en/question/2641/convert-to-command-line-parameter/

we can use that command to convert all documents to odf format (which is an xml)

and then from that format scrape the required data (using BeautifulSoup or something similar)

sample use cases for this process:
- parse plenum protocols -
relevant code: https://github.com/hasadna/Open-Knesset/blob/master/plenum/management/commands/parse_plenum_protocols_subcommands/download.py
(ignore the html parsing parts - concentrate only on the parsing of the xml received from antiword)
- get committee meeting protocol -
relevant code: https://github.com/hasadna/Open-Knesset/blob/master/simple/management/commands/syncdata.py
get_committee_protocol_text method

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.