codelucas / codelucas/newspaper
Problems with finding more uncommon feed urls
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.2k
- Forks
- 2.1k
- PR merge metrics
- No merged PRs in 30d
Description
I have tried newspaper for finding news websites and blog feed urls using Source.set_feeds -
https://github.com/codelucas/newspaper/blob/c0eed1a571ab91382ca7e86767b2c26e7e59bbd2/newspaper/source.py#L128-L132
but I find that it does not deal very well with more uncommon feed urls.
I.E. https://chicagotonight.wttw.com/ uses feedburner for generating it's RSS feed which is not detected by set_feeds' logic.
For this reason I've used feedfinder2 in my projects.
We can build either feedfinder2 or aio-feedfinder2 into newspaper's feed url detection if everyone is okay with how those handle that.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in newspaper/source.py around lines 128-132, where set_feeds detects feed URLs. Compare feedfinder2 and aio-feedfinder2, then verify the approach against the Chicago Tonight FeedBurner URL and existing feed detection behavior. Done means uncommon feed URLs are detected without breaking current cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100