MAINT: bs4 warning during parsing the response for various CADC methods
- Dominant language
- Python
- Stars
- 791
- Forks
- 451
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 4
Description
There is a new warning coming from an updated beautifulsoup version for the `get_acces_url` method.
Could you @andamian confirm whether we should switch to an xlm parser or filter on the warning? switching to use lxml I started to see a cookie related error, so I suppose that way should certainly require some more investigation.
```
=================================== FAILURES ===================================
_____________________________ test_get_access_url ______________________________
@patch('astroquery.cadc.core.get_access_url.caps', {})
def test_get_access_url():
# testing implementation of requests.get method:
def get(url, **kwargs):
class ServiceResponse:
def __init__(self):
self.text = 'ivo://cadc.nrc.ca/mytap = http://my.org/mytap'
def raise_for_status(self):
pass
class CapabilitiesResponse:
def __init__(self):
caps_file = data_path('tap_caps.xml')
with open(caps_file, 'r') as infile:
text = infile.read()
self.text = text
def raise_for_status(self):
pass
if url == conf.CADC_REGISTRY_URL:
return ServiceResponse()
else:
return CapabilitiesResponse()
# now use it in testing
with patch.object(cadc_core.requests, 'get', get):
cadc_core.get_access_url.caps = {}
assert 'http://my.org/mytap' == cadc_core.get_access_url('mytap')
> assert 'https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/argus/tables' == \
cadc_core.get_access_url('mytap',
'ivo://ivoa.net/std/VOSI#tables-1.1')
../../.tox/py37-test-oldestdeps/lib/python3.7/site-packages/astroquery/cadc/tests/test_cadctap.py:195:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../.tox/py37-test-oldestdeps/lib/python3.7/site-packages/astroquery/cadc/core.py:804: in get_access_url
soup = BeautifulSoup(response2.text, features="html5lib")
../../.tox/py37-test-oldestdeps/lib/python3.7/site-packages/bs4/__init__.py:329: in __init__
markup, from_encoding, exclude_encodings=exclude_encodings)):
../../.tox/py37-test-oldestdeps/lib/python3.7/site-packages/bs4/builder/_html5lib.py:77: in prepare_markup
DetectsXMLParsedAsHTML.warn_if_markup_looks_like_xml(markup)
../../.tox/py37-test-oldestdeps/lib/python3.7/site-packages/bs4/builder/__init__.py:538: in warn_if_markup_looks_like_xml
cls._warn()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls =
@classmethod
def _warn(cls):
"""Issue a warning about XML being parsed as HTML."""
warnings.warn(
> XMLParsedAsHTMLWarning.MESSAGE, XMLParsedAsHTMLWarning
)
E bs4.builder.XMLParsedAsHTMLWarning: It looks like you're parsing an XML document using an HTML parser. If this really is an HTML document (maybe it's XHTML?), you can ignore or filter this warning. If it's XML, you should know that using an XML parser will be more reliable. To parse this document as XML, make sure you have the lxml package installed, and pass the keyword argument `features="xml"` into the BeautifulSoup constructor.
../../.tox/py37-test-oldestdeps/lib/python3.7/site-packages/bs4/builder/__init__.py:546: XMLParsedAsHTMLWarning
=========================== short test summary info ============================
FAILED ../../.tox/py37-test-oldestdeps/lib/python3.7/site-packages/astroquery/cadc/tests/test_cadctap.py::test_get_access_url
=========== 1 failed, 1193 passed, 680 skipped, 2 xfailed in 51.96s ============
```
Contributor guide
Research direction
Start with astroquery/cadc/core.py at get_access_url and reproduce the warning using cadc/tests/test_cadctap.py::test_get_access_url. Compare the BeautifulSoup parser choice with the reported cookie-related behavior, then establish a change that removes the warning while keeping the test's expected access URLs and passing the relevant test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100