codelucas / codelucas/newspaper
Parsing Incorrectly for Yahoo Finance
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.2k
- Forks
- 2.1k
- PR merge metrics
- No merged PRs in 30d
Description
for article links like
| url | action required to view all |
|---|---|
| https://finance.yahoo.com/m/ca7cfbea-6f81-34b7-a482-1f7723376eff/ford-crushes-q3-views-with.html | Read More button |
| https://finance.yahoo.com/news/epa-data-shows-tesla-excels-200035538.html | |
| https://finance.yahoo.com/news/q3-gdp-gross-domestic-product-usa-coronavirus-pandemic-181533194.html | Story Continues button to view the remainder of the text |
use case
# url = 'https://finance.yahoo.com/m/ca7cfbea-6f81-34b7-a482-1f7723376eff/ford-crushes-q3-views-with.html' # needs redirect (clicking on Read More)
url = 'https://finance.yahoo.com/news/epa-data-shows-tesla-excels-200035538.html'
from newspaper import Article
a = Article(url)
a.download()
a.parse()
print(a.text[:300]) # attach a debugger
I'm seeing it grab the article text for a suggested article that's down below on the page for the value of a.text.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported cases with newspaper.Article, then run download() and parse() for the listed Yahoo Finance URLs while inspecting a.text. Focus on the article extraction path used by Article; done means the returned text comes from the requested article rather than a suggested article, including for pages with Read More or Story Continues controls.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100