astropy / astropy/astroquery

MAINT: bs4 warning during parsing the response for various CADC methods

Open
#2,523 2 comments 0 reactions 0 assignees View on GitHub
cadc
Dominant language
Python
Stars
791
Forks
451
Avg merge
1d 3h
Merged PRs (30d)
4

Description

There is a new warning coming from an updated beautifulsoup version for the `get_acces_url` method.

Could you @andamian confirm whether we should switch to an xlm parser or filter on the warning? switching to use lxml I started to see a cookie related error, so I suppose that way should certainly require some more investigation.

```
=================================== FAILURES ===================================
_____________________________ test_get_access_url ______________________________

@patch('astroquery.cadc.core.get_access_url.caps', {})
def test_get_access_url():
# testing implementation of requests.get method:
def get(url, **kwargs):
class ServiceResponse:
def __init__(self):
self.text = 'ivo://cadc.nrc.ca/mytap = http://my.org/mytap'

def raise_for_status(self):
pass

class CapabilitiesResponse:
def __init__(self):
caps_file = data_path('tap_caps.xml')
with open(caps_file, 'r') as infile:
text = infile.read()
self.text = text

def raise_for_status(self):
pass
if url == conf.CADC_REGISTRY_URL:
return ServiceResponse()
else:
return CapabilitiesResponse()

# now use it in testing
with patch.object(cadc_core.requests, 'get', get):
cadc_core.get_access_url.caps = {}
assert 'http://my.org/mytap' == cadc_core.get_access_url('mytap')
> assert 'https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/argus/tables' == \
cadc_core.get_access_url('mytap',
'ivo://ivoa.net/std/VOSI#tables-1.1')

../../.tox/py37-test-oldestdeps/lib/python3.7/site-packages/astroquery/cadc/tests/test_cadctap.py:195:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../.tox/py37-test-oldestdeps/lib/python3.7/site-packages/astroquery/cadc/core.py:804: in get_access_url
soup = BeautifulSoup(response2.text, features="html5lib")
../../.tox/py37-test-oldestdeps/lib/python3.7/site-packages/bs4/__init__.py:329: in __init__
markup, from_encoding, exclude_encodings=exclude_encodings)):
../../.tox/py37-test-oldestdeps/lib/python3.7/site-packages/bs4/builder/_html5lib.py:77: in prepare_markup
DetectsXMLParsedAsHTML.warn_if_markup_looks_like_xml(markup)
../../.tox/py37-test-oldestdeps/lib/python3.7/site-packages/bs4/builder/__init__.py:538: in warn_if_markup_looks_like_xml
cls._warn()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cls =

@classmethod
def _warn(cls):
"""Issue a warning about XML being parsed as HTML."""
warnings.warn(
> XMLParsedAsHTMLWarning.MESSAGE, XMLParsedAsHTMLWarning
)
E bs4.builder.XMLParsedAsHTMLWarning: It looks like you're parsing an XML document using an HTML parser. If this really is an HTML document (maybe it's XHTML?), you can ignore or filter this warning. If it's XML, you should know that using an XML parser will be more reliable. To parse this document as XML, make sure you have the lxml package installed, and pass the keyword argument `features="xml"` into the BeautifulSoup constructor.

../../.tox/py37-test-oldestdeps/lib/python3.7/site-packages/bs4/builder/__init__.py:546: XMLParsedAsHTMLWarning
=========================== short test summary info ============================
FAILED ../../.tox/py37-test-oldestdeps/lib/python3.7/site-packages/astroquery/cadc/tests/test_cadctap.py::test_get_access_url
=========== 1 failed, 1193 passed, 680 skipped, 2 xfailed in 51.96s ============
```

Contributor guide

Open the contributing guide

Research direction

Start with astroquery/cadc/core.py at get_access_url and reproduce the warning using cadc/tests/test_cadctap.py::test_get_access_url. Compare the BeautifulSoup parser choice with the reported cookie-related behavior, then establish a change that removes the warning while keeping the test's expected access URLs and passing the relevant test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.