Parsing takes minutes (Raspberry Pi)

What steps will reproduce the problem?

1. Run following python program with any of the feeds listed. Uncomment feed to 
test

#--------start source
import feedparser

print "Parse"
# this one is fast, takes about 8 seconds
#feed = feedparser.parse("http://www.engadget.com/rss.xml")
# next one is about 60 seconds
#feed = feedparser.parse("http://www.eevblog.com/feed/")

# these are the slow ones. Takes minutes to parse:
#feed = feedparser.parse("http://www.heise.de/newsticker/heise-atom.xml")
#feed = feedparser.parse("http://www.spiegel.de/schlagzeilen/tops/index.rss ")


print "Done Parse"
size = len(feed['entries'])
for i in range(0,size):
    print feed['entries'][i].title
#-------- end source



What is the expected output? What do you see instead?
Output is as expected, parsing takes more than 4 minutes per feed.

What version of the product are you using? On what operating system?
Raspberry Pi with standard image
Python 2.7.3
feedparser 5.1.3
BeautifulSoup 3.2.1 (and bs4 installed)

Please provide any additional information below.

Parsing is extremely slow. Seems not to be an encoding issue, as eevblog-feed 
is slow too. Not as slow as German feeds, but slow.
I have included an output from running "python myprogram.py --mcProfile" with 
the "www.heise.de"-feed

These following lines seem to use the most time:
  1538704   48.838    0.000   52.868    0.000 codingstatemachine.py:40(next_state)
      284   31.191    0.110   56.701    0.200 mbcharsetprober.py:52(feed)
     1425   87.178    0.061  115.210    0.081 sbcharsetprober.py:63(feed)
       95   18.991    0.200   42.172    0.444 utf8prober.py:50(feed)

Running the above command did result in errors:
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/cProfile.py", line 199, in <module>
    main()
  File "/usr/lib/python2.7/cProfile.py", line 192, in main
    runctx(code, globs, None, options.outfile, options.sort)
  File "/usr/lib/python2.7/cProfile.py", line 49, in runctx
    prof = prof.runctx(statement, globals, locals)
  File "/usr/lib/python2.7/cProfile.py", line 140, in runctx
    exec cmd in globals, locals
  File "feedreader.py", line 14, in <module>
    print feed['entries'][i].title
UnicodeEncodeError: 'ascii' codec can't encode character u'\xdf' in position 
19: ordinal not in range(128)

Perhaps this helps in figuring out what went wrong. I've searched for days now 
and still don't have a clue...

Hopefully my description makes sense to anyone... On my Mac the program runs 
extremely fast.

Original issue reported on code.google.com by drthomas...@googlemail.com on 16 Nov 2013 at 9:22

Attachments:

• check3.txt

Start by reproducing the reported timings with the listed feeds and review the attached check3.txt profile. Then inspect codingstatemachine.py, mbcharsetprober.py, sbcharsetprober.py, and utf8prober.py, which the profile identifies as hotspots. Done means the feeds parse without multi-minute delays while preserving the expected entries and decoding behavior.