JustAnotherArchivist / JustAnotherArchivist/snscrape
Mastodon: Crash when attempting to retrieve data from Mastodon servers running v4.0.0
- Dominant language
- Python
- Stars
- 5.4k
- Forks
- 783
- PR merge metrics
- No merged PRs in 30d
Description
Crash occurs when attempting to retrieve data from a user profile at mastodon.social (v4.0.0rc1) or mstdn.social (v4.0.0rc1) while using snscrape 0.4.3.20220107.dev64+g59abeaf
Crash does not occur when retrieving data from servers running earlier versions of Mastodon (3.x.x)
Doing so yields the following stack trace:
```
Traceback (most recent call last):
File "[path-to-lib]\site-packages\snscrape\_cli.py", line 116, in _dump_locals_on_exception
yield
File "[path-to-lib]\site-packages\snscrape\_cli.py", line 318, in main
for i, item in enumerate(scraper.get_items(), start = 1):
File "[path-to-lib]\site-packages\snscrape\modules\mastodon.py", line 279, in get_items
yield from self._entries_to_items(soup.find('div', class_ = 'activity-stream').find_all('div', class_ = 'entry'), r.url)
AttributeError: 'NoneType' object has no attribute 'find_all'
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in snscrape/modules/mastodon.py at get_items(), where the stack trace shows activity-stream is looked up before find_all(). Reproduce profile retrieval against a Mastodon 4.0.0 server and compare it with a 3.x server; done means retrieval no longer crashes on the newer response while existing 3.x behavior remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100