Scanner fails to visit the next site when something goes wrong extracting data from PB's options page
- Dominant language
- Python
- Stars
- 143
- Forks
- 19
- PR merge metrics
- No merged PRs in 30d
Description
We could error out at the following point with a JavascriptException (some WebDriverException) or a TimeoutException:
https://github.com/EFForg/badger-sett/blob/9f7da4b3ae60400e5b7f2c096f77b9de02e2d6d2/crawler.py#L1111
If this happens, we:
- Don't log "Visiting ..." for the site
- Fail to visit the site
- Confusingly log the error that happened dumping data as if it happened visiting the site (instead of our extension page)
To find scans where this happened:
```bash
sqlite3 -batch badger.sqlite3 'select * from scan s join (select scan_id, count(*) num_attempts from scan_sites join scan on scan.id = scan_sites.scan_id where strftime("%s", scan.start_time) > strftime("%s", "2025-01-08 13:00:56") group by scan_id having num_attempts != scan.num_sites) sub on sub.scan_id = s.id order by start_time asc'
```
To find occurrences of this bug within a given scan (update hash and total expected site count):
```bash
(tmpfile=$(mktemp) && git show 660a07c:log.txt > "$tmpfile" && for i in $(seq 1 6600); do grep -q "Visiting ${i}:" "$tmpfile" || echo "${i}: nope"; done; rm "$tmpfile")
```
I noticed this when the scan db updater script failed (fix: 83a33af21bb675dce0d741e668d395f8c0ed964f) and there was a skipped site in the scan log following the exception.
I don't think the fix is to move the "visiting" line up (previously: fa35217b).
I think we should add retrying/restarting to dump_data. Would also help with failing the scan at the very end here (actually happens sometimes):
https://github.com/EFForg/badger-sett/blob/9f7da4b3ae60400e5b7f2c096f77b9de02e2d6d2/crawler.py#L1164
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.