Flaky stream load balancing
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 24
- Forks
- 16
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 1
Description
From time to time, the stream is not load balanced correctly between Mist nodes. I attach the logs of both catalyst nodes.
[logs-node-1.txt](https://github.com/livepeer/catalyst/files/9221102/logs-node-1.txt)
[logs-node-2.txt](https://github.com/livepeer/catalyst/files/9221103/logs-node-2.txt)
### Steps to reproduce
#### 1. Check out the code from the branch with additional debug logs
Check out the branch [`reproduce-load-balancing-issue`](https://github.com/livepeer/catalyst/tree/rafal/reproduce-load-balancing-issue). It adds additional logs and increases the timeout of the E2E Test
#### 2. Run many tests sequentially to reproduce the flakiness
```
for i in $(seq 0 30); do go test -timeout 30m -v ./test/e2e/*; done
```
#### 3. Notice that a test was stuck
You should see a message that the test is stuck.
```
#####
STUCK in requireReplicatedStream
```
#### 4. Check the container logs
Now, you can check the logs of the containers and investigate the issue.
```
docker ps
```
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Check out the reproduce-load-balancing-issue branch and run the repeated E2E command in test/e2e. Start with the requireReplicatedStream stuck message, then inspect the attached logs and Docker container logs. Done means identifying the cause of the incorrect stream distribution between Mist nodes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100