google / google/corpuscrawler

Undefined names

Open
#87 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
218
Forks
53
PR merge metrics
No merged PRs in 30d

Description

% __flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics__
```
./corpuscrawler/Lib/corpuscrawler/crawl_mi.py:62:39: F821 undefined name 'sitemap'
if pubdate is None: pubdate = sitemap[url]
^
./corpuscrawler/Lib/corpuscrawler/crawl_kab.py:53:48: F821 undefined name 'url'
assert doc.status == 200, (doc.status, url)
^
./corpuscrawler/Lib/corpuscrawler/crawl_tpi.py:48:48: F821 undefined name 'url'
assert doc.status == 200, (doc.status, url)
^
./corpuscrawler/Lib/corpuscrawler/crawl_shn.py:90:30: F821 undefined name 'striptags'
p = ' '.join(striptags(replace_html_entities(p)).split())
^
./corpuscrawler/Lib/corpuscrawler/crawl_shn.py:90:40: F821 undefined name 'replace_html_entities'
p = ' '.join(striptags(replace_html_entities(p)).split())
^
./corpuscrawler/Lib/corpuscrawler/crawl_ga.py:147:39: F821 undefined name 'fetchresult'
if pubdate is None: pubdate = fetchresult.headers.get('Last-Modified')
^
./corpuscrawler/Lib/corpuscrawler/crawl_th.py:25:5: F821 undefined name 'crawl_bibleis'
crawl_bibleis(crawler, out, bible='THATSV')
^
./corpuscrawler/Lib/corpuscrawler/crawl_vec.py:43:48: F821 undefined name 'start_url'
assert doc.status == 200, (doc.status, start_url)
^
8 F821 undefined name 'fetchresult'
8
```
https://flake8.pycqa.org/en/latest/user/error-codes.html

On the flake8 test selection, this PR does _not_ focus on "_style violations_" (the majority of flake8 error codes that [__psf/black__](https://github.com/psf/black) can autocorrect). Instead, these tests are focus on runtime safety and correctness:
* E9 tests are about Python syntax errors usually raised because flake8 can not build an Abstract Syntax Tree (AST). Often these issues are a sign of unused code or code that has not been ported to Python 3. These would be compile-time errors in a compiled language but in a dynamic language like Python, they result in the script halting/crashing on the user.
* F63 tests are usually about the confusion between identity and equality in Python. Use ==/!= to compare str, bytes, and int literals is the classic case. These are areas where __a == b__ is True but __a is b__ is False (or vice versa). Python >= 3.8 will raise SyntaxWarnings on these instances.
* F7 tests logic errors and syntax errors in type hints
* F82 tests are almost always _undefined names_ which are usually a sign of a typo, missing imports, or code that has not been ported to Python 3. These also would be compile-time errors in a compiled language but in Python, a __NameError__ is raised which will halt/crash the script on the user.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.