EFForg / EFForg/badger-sett

Scanner fails to visit the next site when something goes wrong extracting data from PB's options page

Open
#99 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
143
Forks
19
PR merge metrics
No merged PRs in 30d

Description

We could error out at the following point with a JavascriptException (some WebDriverException) or a TimeoutException:

https://github.com/EFForg/badger-sett/blob/9f7da4b3ae60400e5b7f2c096f77b9de02e2d6d2/crawler.py#L1111

If this happens, we:

- Don't log "Visiting ..." for the site
- Fail to visit the site
- Confusingly log the error that happened dumping data as if it happened visiting the site (instead of our extension page)

To find scans where this happened:
```bash
sqlite3 -batch badger.sqlite3 'select * from scan s join (select scan_id, count(*) num_attempts from scan_sites join scan on scan.id = scan_sites.scan_id where strftime("%s", scan.start_time) > strftime("%s", "2025-01-08 13:00:56") group by scan_id having num_attempts != scan.num_sites) sub on sub.scan_id = s.id order by start_time asc'
```

To find occurrences of this bug within a given scan (update hash and total expected site count):
```bash
(tmpfile=$(mktemp) && git show 660a07c:log.txt > "$tmpfile" && for i in $(seq 1 6600); do grep -q "Visiting ${i}:" "$tmpfile" || echo "${i}: nope"; done; rm "$tmpfile")
```

I noticed this when the scan db updater script failed (fix: 83a33af21bb675dce0d741e668d395f8c0ed964f) and there was a skipped site in the scan log following the exception.

I don't think the fix is to move the "visiting" line up (previously: fa35217b).

I think we should add retrying/restarting to dump_data. Would also help with failing the scan at the very end here (actually happens sometimes):

https://github.com/EFForg/badger-sett/blob/9f7da4b3ae60400e5b7f2c096f77b9de02e2d6d2/crawler.py#L1164

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.