JustAnotherArchivist / JustAnotherArchivist/snscrape
10-20 times lower throughput of using SNScrape on a container in comparison to non containerized environment
- Dominant language
- Python
- Stars
- 5.4k
- Forks
- 783
- PR merge metrics
- No merged PRs in 30d
Description
### Describe the bug
I am not sure what exactly could be wrong here but I thought to share my experience here in case anybody else faced the same issue.
Using SNScrape locally on a non-containerized environment takes about 0.2-0.3s on fetching tweets on average which is pretty good for our use case. However, when the same code runs on a container it causes the latency to be significantly higher (e.g. 3-5s). We thought perhaps the CPU throttling causing issues here or it was some sort of issue with warming up the container as we have been using SNScrape via the command line, so we attempted to test it as a service and the same issue persists. We have attempted to test it on Minikube and Kubernetes and the same issue persists. Below is the outcome of the profiling we did to understand what could be wrong here.
Profiling on a container:
https://justpaste.it/alqiq
Profiling on a non-containerized env:
https://justpaste.it/3frkx
Interestingly even running `snscrape --version` in a container is super slow!
My guess is we are using some libraries in SNscrape that is much faster in a non-containerized mode in comparison to when they run on a container. We haven't tested different base docker images yet.
### How to reproduce
Run it on a container. SSH to the container and try `snscrape --version`
### Expected behaviour
Expect to have relatively similar throughput when running SNScrape on a container vs non-containerized env.
### Screenshots and recordings
_No response_
### Operating system
Alpine Linux v3.12
### Python version: output of `python3 --version`
3.8.10
### snscrape version: output of `snscrape --version`
0.6.2.20230320
### Scraper
twitter-user
### How are you using snscrape?
CLI (`snscrape ...` as a command, e.g. in a terminal)
### Backtrace
_No response_
### Log output
_No response_
### Dump of locals
_No response_
### Additional context
_No response_
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing `snscrape --version` and the `twitter-user` command in the reported Alpine Linux 3.12 container and a non-containerized environment. Compare the linked container and non-container profiling results, then identify and resolve the source of the latency difference; done means comparable throughput or a documented, reproducible limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, kubernetes, python
- Domain
- cli, devops, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100