scrapinghub / scrapinghub/dateparser
Dataset of article publication dates as they appear on the web
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 520
- Avg merge
- 22h 56m
- Merged PRs (30d)
- 6
Description
Hi, on behalf of Automatic Extraction team from Zyte, I'd like to thank dateparser developers for a great library, and share a dataset of article publication dates as they appear on the web - I think it can be useful for the library development. It was collected on Oct 2020 - Mar 2021, has 300k rows, and contains the following columns:
suffixis the TLD (top level domain), e.g.comorru, extracted withtldextract.extract(domain).suffixarticleLanguageis the language of the article (article.inLanguagefield from https://docs.zyte.com/automatic-extraction/article.html)webPageLanguagesis a space separated list of web page languages (webPage.inLanguagesfrom https://docs.zyte.com/automatic-extraction/article.html)datePublishedRawis the publication date of the article as it appeared on the web-site (article.datePublishedRawfrom https://docs.zyte.com/automatic-extraction/article.html) - non-empty.
Here are the first few rows:
| suffix | articleLanguage | webPageLanguages | datePublishedRaw |
|---|---|---|---|
| eu | en | en lg | October 22, 2020 |
| es | es | es ca | 2020-10-09T17:53:37+02:00 |
| fr | fr | fr en | 02/10/2020 14:53:07 |
| ie | en | en | 2020-10-19T12:54:52Z |
| com | en | en th | 28 Oct 2020 at 13:50 |
And here it dataset in full: article_date_sample_Oct_2020_Mar_2021_public.csv.zip
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the attached article_date_sample_Oct_2020_Mar_2021_public.csv.zip and the listed dataset columns. The issue does not name a target file, test, or concrete code change; completion would require deciding how, if at all, this 300k-row dataset should support dateparser development.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100