google / google/civics_cdf_validator
Performance on large election feeds
- Dominant language
- Python
- Stars
- 38
- Forks
- 23
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 8
Description
Hi, just wanted to drop a note that we're experiencing very long run times when running the validator on large election feeds. This doesn't seem to be a problem with officeholder feeds. For example, the validator processes a ~25MB, ~600K lines officeholders feed in just 40 seconds. But a ~67MB, ~1.6M lines elections feed takes nearly 40 minutes (and my computer is running at nearly 100% CPU the whole time).
Someone mentioned an available flag for profiling the validator, but I'm not seeing anything about that in the README or codebase. If you can point me to docs about it, I'll re-run our elections feed with profiling to try to narrow down what's taking so long.
As you can imagine, having the validator take so long impacts our ability to do iterative development and debugging of feeds, so anything that can be done to improve the performance would be most appreciated. Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.