JustAnotherArchivist / JustAnotherArchivist/snscrape

Mastodon: Crash when attempting to retrieve data from Mastodon servers running v4.0.0

Open
#587 1 comment 1 reaction 0 assignees View on GitHub
bug module:mastodon
Dominant language
Python
Stars
5.4k
Forks
783
PR merge metrics
No merged PRs in 30d

Description

Crash occurs when attempting to retrieve data from a user profile at mastodon.social (v4.0.0rc1) or mstdn.social (v4.0.0rc1) while using snscrape 0.4.3.20220107.dev64+g59abeaf

Crash does not occur when retrieving data from servers running earlier versions of Mastodon (3.x.x)

Doing so yields the following stack trace:

```
Traceback (most recent call last):
File "[path-to-lib]\site-packages\snscrape\_cli.py", line 116, in _dump_locals_on_exception
yield
File "[path-to-lib]\site-packages\snscrape\_cli.py", line 318, in main
for i, item in enumerate(scraper.get_items(), start = 1):
File "[path-to-lib]\site-packages\snscrape\modules\mastodon.py", line 279, in get_items
yield from self._entries_to_items(soup.find('div', class_ = 'activity-stream').find_all('div', class_ = 'entry'), r.url)
AttributeError: 'NoneType' object has no attribute 'find_all'
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in snscrape/modules/mastodon.py at get_items(), where the stack trace shows activity-stream is looked up before find_all(). Reproduce profile retrieval against a Mastodon 4.0.0 server and compare it with a 3.x server; done means retrieval no longer crashes on the newer response while existing 3.x behavior remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.