JustAnotherArchivist / JustAnotherArchivist/snscrape
Reddit scraper returns no submissions before 2022-11-03
- Dominant language
- Python
- Stars
- 5.4k
- Forks
- 783
- PR merge metrics
- No merged PRs in 30d
Description
### Describe the bug
When I scrape reddits for November 2022 onward, I get 10 fields:
[_type', 'author', 'body', 'date', 'id', 'parentId', 'subreddit', 'url', 'link', 'selftext', 'title']
When I scrape reddits before November 2022 (more than 3 months ago), I only get 7 fields:
['_type', 'author', 'body', 'date', 'id', 'parentId', 'subreddit', 'url']
I am missing the last 3 fields for reddits that date before November 2022: "title", "selftext", and "link"
When I scraped reddits last year (in early December), I got all 10 fields even for reddits from 10 years ago.
### How to reproduce
**October 2022**
**_In:_**
```
!snscrape --json --progress reddit-search --before 1667275199 --after 1664596800 Searchterm > October.json
df = pd.read_json("October.json", lines=True)
df.columns.tolist()
```
_**Out:**_
['_type', 'author', 'body', 'date', 'id', 'parentId', 'subreddit', 'url']
**Nov 2022**
**_In:_**
```
!snscrape --json --progress reddit-search --before 1669870799 --after 1667275200 Searchterm > November.json
df = pd.read_json("November.json", lines=True)
df.columns.tolist()
```
_**Out:**_
['_type', 'author', 'body', 'date', 'id', 'parentId', 'subreddit', 'url', 'link', 'selftext', 'title']
### Expected behavior
All 10 fields are scraped regardless of reddit date.
### Screenshots and recordings
_No response_
### OS / Distro
Ubuntu 18.04.6 LTS
### Output from `snscrape --version`
snscrape 0.5.0.20230113
### Scraper
reddit-search
### Backtrace
_No response_
### Dump of locals
_No response_
### How are you using snscrape?
CLI
### Additional context
Called from python notebook. Same outcome if called directly from command line.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.