JustAnotherArchivist / JustAnotherArchivist/snscrape

Reddit scraper returns no submissions before 2022-11-03

Open
#689 1 comment 0 reactions 0 assignees View on GitHub
bug module:reddit upstream
Dominant language
Python
Stars
5.4k
Forks
783
PR merge metrics
No merged PRs in 30d

Description

### Describe the bug

When I scrape reddits for November 2022 onward, I get 10 fields:
[_type', 'author', 'body', 'date', 'id', 'parentId', 'subreddit', 'url', 'link', 'selftext', 'title']

When I scrape reddits before November 2022 (more than 3 months ago), I only get 7 fields:
['_type', 'author', 'body', 'date', 'id', 'parentId', 'subreddit', 'url']

I am missing the last 3 fields for reddits that date before November 2022: "title", "selftext", and "link"

When I scraped reddits last year (in early December), I got all 10 fields even for reddits from 10 years ago.

### How to reproduce

**October 2022**
**_In:_**
```
!snscrape --json --progress reddit-search --before 1667275199 --after 1664596800 Searchterm > October.json
df = pd.read_json("October.json", lines=True)
df.columns.tolist()
```
_**Out:**_
['_type', 'author', 'body', 'date', 'id', 'parentId', 'subreddit', 'url']

**Nov 2022**
**_In:_**
```
!snscrape --json --progress reddit-search --before 1669870799 --after 1667275200 Searchterm > November.json
df = pd.read_json("November.json", lines=True)
df.columns.tolist()
```
_**Out:**_
['_type', 'author', 'body', 'date', 'id', 'parentId', 'subreddit', 'url', 'link', 'selftext', 'title']

### Expected behavior

All 10 fields are scraped regardless of reddit date.

### Screenshots and recordings

_No response_

### OS / Distro

Ubuntu 18.04.6 LTS

### Output from `snscrape --version`

snscrape 0.5.0.20230113

### Scraper

reddit-search

### Backtrace

_No response_

### Dump of locals

_No response_

### How are you using snscrape?

CLI

### Additional context

Called from python notebook. Same outcome if called directly from command line.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.