jmcarp / jmcarp/robobrowser

badly handle web pages with encoding errors

Open
#58 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.7k
Forks
336
PR merge metrics
No merged PRs in 30d

Description

As the BeautifullSoup doc says : 'Sometimes a document is mostly in UTF-8, but contains Windows-1252 characters...' . As an example, all pages from the 'http://www.programme-tv.com' web site are like this (due to the Ô in 'France Ô').
In that case, RoboBrowser convert the full document to Windows-1252 encoding, leaving all accentuated chars unreadable :

``` python
from robobrowser import RoboBrowser
browser = RoboBrowser()
browser.open('http://www.programme-tv.com')
browser.find('span', 'slogan1').text
```

The output is :
`'Ne ratez plus vos émissions favorites!'`
instead of
`'Ne ratez plus vos Émissions favorites!'`

Using self.response.text instead of self.response.content when calling BeautifullSoup is solving the problem but it probably have some drawbacks.

Cheers,
Loïc

Contributor guide

Open the contributing guide

Research direction

Start at RoboBrowser.open and the call that passes self.response.content to BeautifulSoup; compare that path with the reported self.response.text alternative. Reproduce the programme-tv.com example and confirm that the slogan retains readable accented characters without breaking other response handling.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.