internetarchive / internetarchive/openlibrary-bots

Large ONIX Files require iterparse (else out-of-memory errors)

Open
#17 0 comments 0 reactions 0 assignees View on GitHub
ONIX bot requirement
Dominant language
Python
Stars
81
Forks
67
PR merge metrics
No merged PRs in 30d

Description

lxml has an option for iteratively parsing xml files which we will want to implement, otherwise, we will run out of memory. Here's an example:

https://lxml.de/parsing.html#iterparse-and-iterwalk

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.