livepeer / livepeer/catalyst

MistContoller fails to start other components

Open
#67 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go
Stars
24
Forks
16
Avg merge
1d 8h
Merged PRs (30d)
1

Description

There is some race condition, which makes the execution of the `livepeer/catalyst` container flaky. In result, sometimes, when catalyst is started, there are no logs at all and the only running process is `MistController`.

### Steps to reproduce

#### Start multiple livepeer catalyst and find the ones with no logs

```
for i in $(seq 0 100); do CID=$(docker run -d --rm livepeer/catalyst); sleep 5; LOGS=$(docker logs $CID); [[ -z "$LOGS" ]] || docker kill $CID;done
```

This command will start 100 container one by one and kill all expect the ones that produce no logs. Then check the running containers and if there are none, then retry the command above. In my case, there was always at least one running container left.

```
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
79e3ec38e0c6 livepeer/catalyst "/usr/bin/MistContro…" 8 minutes ago Up 8 minutes 1935/tcp, 4242/tcp, 8080/tcp, 8889/udp sleepy_poincare
```

#### 2. Exec into the container and check the running processes

You should find that the only running process is `MistController`.

```
$ docker exec -it 79e3ec38e0c6 sh
# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 16080 4340 ? Ssl 13:47 0:00 /usr/bin/MistController -c /etc/livepeer/catalyst.json
root 14 0.1 0.0 2060 496 pts/0 Ss 13:57 0:00 sh
root 20 0.0 0.0 5480 2384 pts/0 R+ 13:57 0:00 ps aux
```

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with the provided docker run loop, then inspect the container entry point /usr/bin/MistController and configuration at /etc/livepeer/catalyst.json. Check which component startup step can leave only MistController running. Done means repeated container starts produce logs and all expected components remain running.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
backend, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.