codelucas / codelucas/newspaper
newspaper cannot parse or encode error
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.2k
- Forks
- 2.1k
- PR merge metrics
- No merged PRs in 30d
Description
Hi, I am trying to crawl Korean news papers.
Most news articles crawled and parsed well
but some articles such as below url, newspaper cannot parse well in Korean.
http://www.edaily.co.kr/news/newspath.asp?newsid=01610486605953456
After parsing the article, the result is as below the picture.
(Parsed title should be "금감원, 소비자단체와 금융현장서 민원 상담·구제")
As shown in the a.title, I think newspaper have an encoding problem.
What do you think of this problem?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported URL and inspect how newspaper parses and decodes its title. Compare the result with the expected Korean title "금감원, 소비자단체와 금융현장서 민원 상담·구제"; done means this article's title is parsed correctly without breaking the other articles that already work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100