badly handle web pages with encoding errors
- Dominant language
- Python
- Stars
- 3.7k
- Forks
- 336
- PR merge metrics
- No merged PRs in 30d
Description
As the BeautifullSoup doc says : 'Sometimes a document is mostly in UTF-8, but contains Windows-1252 characters...' . As an example, all pages from the 'http://www.programme-tv.com' web site are like this (due to the Ô in 'France Ô').
In that case, RoboBrowser convert the full document to Windows-1252 encoding, leaving all accentuated chars unreadable :
``` python
from robobrowser import RoboBrowser
browser = RoboBrowser()
browser.open('http://www.programme-tv.com')
browser.find('span', 'slogan1').text
```
The output is :
`'Ne ratez plus vos émissions favorites!'`
instead of
`'Ne ratez plus vos Émissions favorites!'`
Using self.response.text instead of self.response.content when calling BeautifullSoup is solving the problem but it probably have some drawbacks.
Cheers,
Loïc
Contributor guide
Research direction
Start at RoboBrowser.open and the call that passes self.response.content to BeautifulSoup; compare that path with the reported self.response.text alternative. Reproduce the programme-tv.com example and confirm that the slogan retains readable accented characters without breaking other response handling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100