JustAnotherArchivist / JustAnotherArchivist/snscrape

10-20 times lower throughput of using SNScrape on a container in comparison to non containerized environment

Open
#946 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.4k
Forks
783
PR merge metrics
No merged PRs in 30d

Description

### Describe the bug

I am not sure what exactly could be wrong here but I thought to share my experience here in case anybody else faced the same issue.

Using SNScrape locally on a non-containerized environment takes about 0.2-0.3s on fetching tweets on average which is pretty good for our use case. However, when the same code runs on a container it causes the latency to be significantly higher (e.g. 3-5s). We thought perhaps the CPU throttling causing issues here or it was some sort of issue with warming up the container as we have been using SNScrape via the command line, so we attempted to test it as a service and the same issue persists. We have attempted to test it on Minikube and Kubernetes and the same issue persists. Below is the outcome of the profiling we did to understand what could be wrong here.

Profiling on a container:
https://justpaste.it/alqiq

Profiling on a non-containerized env:
https://justpaste.it/3frkx

Interestingly even running `snscrape --version` in a container is super slow!

My guess is we are using some libraries in SNscrape that is much faster in a non-containerized mode in comparison to when they run on a container. We haven't tested different base docker images yet.

### How to reproduce

Run it on a container. SSH to the container and try `snscrape --version`

### Expected behaviour

Expect to have relatively similar throughput when running SNScrape on a container vs non-containerized env.

### Screenshots and recordings

_No response_

### Operating system

Alpine Linux v3.12

### Python version: output of `python3 --version`

3.8.10

### snscrape version: output of `snscrape --version`

0.6.2.20230320

### Scraper

twitter-user

### How are you using snscrape?

CLI (`snscrape ...` as a command, e.g. in a terminal)

### Backtrace

_No response_

### Log output

_No response_

### Dump of locals

_No response_

### Additional context

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing `snscrape --version` and the `twitter-user` command in the reported Alpine Linux 3.12 container and a non-containerized environment. Compare the linked container and non-container profiling results, then identify and resolve the source of the latency difference; done means comparable throughput or a documented, reproducible limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, kubernetes, python
Domain
cli, devops, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.