google / google/civics_cdf_validator

Performance on large election feeds

Open
#244 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
38
Forks
23
Avg merge
1d 12h
Merged PRs (30d)
8

Description

Hi, just wanted to drop a note that we're experiencing very long run times when running the validator on large election feeds. This doesn't seem to be a problem with officeholder feeds. For example, the validator processes a ~25MB, ~600K lines officeholders feed in just 40 seconds. But a ~67MB, ~1.6M lines elections feed takes nearly 40 minutes (and my computer is running at nearly 100% CPU the whole time).

Someone mentioned an available flag for profiling the validator, but I'm not seeing anything about that in the README or codebase. If you can point me to docs about it, I'll re-run our elections feed with profiling to try to narrow down what's taking so long.

As you can imagine, having the validator take so long impacts our ability to do iterative development and debugging of feeds, so anything that can be done to improve the performance would be most appreciated. Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.