livepeer / livepeer/go-livepeer

Delayed segment submission errors during O failover

Open
#697 3 comments 0 reactions 1 assignee Claimed by @j0sh View on GitHub
type: bug
Dominant language
Go
Stars
586
Forks
226
Avg merge
1d 17h
Merged PRs (30d)
19

Description

**Describe the bug**
A clear and concise description of what the bug is.

I observed a scenario where a B is sending segments to an O, the O goes offline (stopped the Docker container created by the test harness), but the B appears to continue submitting segments to O. In the logs, all I saw was messages indicating segments were submitted, but no messages indicating an error or a successful transcode. The B also did not also try fetching a new O.

However, when I rebooted that same O, all the error messages from the previous segment submissions were logged by B. Then, B fetched for Os and ended up being assigned to the O that was just rebooted. Afterwords, B continues to submit segments and successfully receives transcoded results.

I observed this behavior once over the course of many runs using a set up of 1 B and 3 Os while testing fail over.

**To Reproduce**
Steps to reproduce the behavior:
1. Start 1 B and 3 Os that are registered on-chain
2. Start streaming into B
3. Try stopping whichever O that B is assigned to
4. Try repeating this process until the described behavior occurs

**Expected behavior**
A clear and concise description of what you expected to happen.

If the O that B is assigned to goes down, B should immediately fail over to another O by fetching for O responses again (if there are no Os available B should at least by fetching for O responses, but it will just end up with 0 responses).

**Screenshots**
If applicable, add screenshots to help explain your problem.

Github won't let me post the screenshot for some reason so here is the copy and pasted version:

```
I0118 20:53:26.079435 1 rpc.go:530] Submitting segment 65 : 559488 bytes
I0118 20:53:28.203446 1 rpc.go:530] Submitting segment 66 : 554412 bytes
I0118 20:53:30.218804 1 rpc.go:530] Submitting segment 67 : 560428 bytes
I0118 20:53:32.240584 1 rpc.go:530] Submitting segment 68 : 555164 bytes
I0118 20:53:34.263659 1 rpc.go:530] Submitting segment 69 : 558172 bytes
I0118 20:53:36.284134 1 rpc.go:530] Submitting segment 70 : 555164 bytes
I0118 20:53:38.297248 1 rpc.go:530] Submitting segment 71 : 563812 bytes
I0118 20:53:40.412267 1 rpc.go:530] Submitting segment 72 : 555916 bytes
I0118 20:53:42.428190 1 rpc.go:530] Submitting segment 73 : 552532 bytes
E0118 20:53:43.031857 1 rpc.go:535] Unable to submit segment 66 Post https://lp_orchestrator_0:8935/segment: dial tcp 192.168.208.3:8935: getsockopt: connection refused (Client.Timeout exceeded while awaiting headers)
E0118 20:53:43.031911 1 rpc.go:535] Unable to submit segment 57 Post https://lp_orchestrator_0:8935/segment: dial tcp 192.168.208.3:8935: getsockopt: connection refused (Client.Timeout exceeded while awaiting headers)
E0118 20:53:43.031946 1 rpc.go:535] Unable to submit segment 56 Post https://lp_orchestrator_0:8935/segment: dial tcp 192.168.208.3:8935: getsockopt: connection refused (Client.Timeout exceeded while awaiting headers)
E0118 20:53:43.031978 1 rpc.go:535] Unable to submit segment 55 Post https://lp_orchestrator_0:8935/segment: dial tcp 192.168.208.3:8935: getsockopt: connection refused (Client.Timeout exceeded while awaiting headers)
E0118 20:53:43.032002 1 rpc.go:535] Unable to submit segment 54 Post https://lp_orchestrator_0:8935/segment: dial tcp 192.168.208.3:8935: getsockopt: connection refused (Client.Timeout exceeded while awaiting headers)
E0118 20:53:43.032026 1 rpc.go:535] Unable to submit segment 53 Post https://lp_orchestrator_0:8935/segment: dial tcp 192.168.208.3:8935: getsockopt: connection refused (Client.Timeout exceeded while awaiting headers)
E0118 20:53:43.032051 1 rpc.go:535] Unable to submit segment 52 Post https://lp_orchestrator_0:8935/segment: dial tcp 192.168.208.3:8935: getsockopt: connection refused (Client.Timeout exceeded while awaiting headers)
E0118 20:53:43.032113 1 rpc.go:535] Unable to submit segment 51 Post https://lp_orchestrator_0:8935/segment: dial tcp 192.168.208.3:8935: getsockopt: connection refused (Client.Timeout exceeded while awaiting headers)
E0118 20:53:43.032151 1 rpc.go:535] Unable to submit segment 69 Post https://lp_orchestrator_0:8935/segment: dial tcp 192.168.208.3:8935: getsockopt: connection refused (Client.Timeout exceeded while awaiting headers)
E0118 20:53:43.032207 1 rpc.go:535] Unable to submit segment 73 Post https://lp_orchestrator_0:8935/segment: dial tcp 192.168.208.3:8935: getsockopt: connection refused
E0118 20:53:43.032232 1 rpc.go:535] Unable to submit segment 72 Post https://lp_orchestrator_0:8935/segment: dial tcp 192.168.208.3:8935: getsockopt: connection refused
```

```
E0118 20:53:43.033689 1 rpc.go:535] Unable to submit segment 61 Post https://lp_orchestrator_0:8935/segment: dial tcp 192.168.208.3:8935: getsockopt: connection refused (Client.Timeout exceeded while awaiting headers)
I0118 20:53:43.032773 1 rpc.go:119] Connecting RPC to https://lp_orchestrator_0:8935
E0118 20:53:43.031883 1 rpc.go:535] Unable to submit segment 50 Post https://lp_orchestrator_0:8935/segment: dial tcp 192.168.208.3:8935: getsockopt: connection refused (Client.Timeout exceeded while awaiting headers)
E0118 20:53:43.033234 1 rpc.go:535] Unable to submit segment 46 Post https://lp_orchestrator_0:8935/segment: dial tcp 192.168.208.3:8935: getsockopt: connection refused (Client.Timeout exceeded while awaiting headers)
I0118 20:53:43.033376 1 rpc.go:119] Connecting RPC to https://lp_orchestrator_2:8935
I0118 20:53:43.033397 1 rpc.go:119] Connecting RPC to https://lp_orchestrator_1:8935
E0118 20:53:43.033423 1 rpc.go:535] Unable to submit segment 44 Post https://lp_orchestrator_0:8935/segment: dial tcp 192.168.208.3:8935: getsockopt: connection refused (Client.Timeout exceeded while awaiting headers)
E0118 20:53:43.033442 1 rpc.go:535] Unable to submit segment 60 Post https://lp_orchestrator_0:8935/segment: dial tcp 192.168.208.3:8935: getsockopt: connection refused (Client.Timeout exceeded while awaiting headers)
E0118 20:53:43.033319 1 rpc.go:535] Unable to submit segment 42 Post https://lp_orchestrator_0:8935/segment: dial tcp 192.168.208.3:8935: getsockopt: connection refused (Client.Timeout exceeded while awaiting headers)
E0118 20:53:43.033618 1 rpc.go:535] Unable to submit segment 43 Post https://lp_orchestrator_0:8935/segment: dial tcp 192.168.208.3:8935: getsockopt: connection refused (Client.Timeout exceeded while awaiting headers)
E0118 20:53:43.033662 1 rpc.go:535] Unable to submit segment 59 Post https://lp_orchestrator_0:8935/segment: dial tcp 192.168.208.3:8935: getsockopt: connection refused (Client.Timeout exceeded while awaiting headers)
E0118 20:53:43.031801 1 rpc.go:535] Unable to submit segment 58 Post https://lp_orchestrator_0:8935/segment: dial tcp 192.168.208.3:8935: getsockopt: connection refused (Client.Timeout exceeded while awaiting headers)
I0118 20:53:45.469987 1 discovery.go:100] Done fetching orch info for orchestrators, numResponses fetched: [transcoder:"https://lp_orchestrator_0:8935" ticket_params: ]
E0118 20:53:46.013742 1 rpc.go:125] Did not connect: context deadline exceeded
E0118 20:53:46.013800 1 rpc.go:125] Did not connect: context deadline exceeded
I0118 20:53:46.035303 1 rpc.go:530] Submitting segment 75 : 565692 bytes
I0118 20:53:46.046248 1 rpc.go:553] Uploaded segment 75
I0118 20:53:47.009617 1 rpc.go:610] Successfully transcoded segment 75
I0118 20:53:48.053592 1 rpc.go:530] Submitting segment 76 : 556292 bytes
I0118 20:53:48.068441 1 rpc.go:553] Uploaded segment 76
I0118 20:53:49.106996 1 rpc.go:610] Successfully transcoded segment 76
I0118 20:53:50.076814 1 rpc.go:530] Submitting segment 77 : 560992 bytes
I0118 20:53:50.090222 1 rpc.go:553] Uploaded segment 77
```

**Desktop (please complete the following information):**
- OS: OSX
- Version: f023c29

Nodes started using the test harness.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.