livepeer / livepeer/go-livepeer

S3 Failure Causes Video To Skip

Open
#841 2 comments 0 reactions 1 assignee View on GitHub

@darkdarkdragon is already working on this.

Since Apr 25, 2019.

area: broadcasting status: icebox type: bug
Dominant language
Go
Stars
586
Forks
226
Avg merge
1d 17h
Merged PRs (30d)
19

Description

**Describe the bug**
When a "save" request to S3 fails, the video stops.

Logs from the O/T:
```
I0424 16:30:09.862917 28004 orchestrator.go:236] Starting to transcode segment 31
I0424 16:30:09.862933 28004 orchestrator.go:245] Submitted segment to transcode loop 31
I0424 16:30:10.483774 28004 orchestrator.go:332] Transcoding of segment 31 took 620.325618ms
I0424 16:30:10.484509 28004 s3.go:131] Saving to S3 https:/livepeer-cbs.s3.amazonaws.com/test1/987df0f8/P360p30fps16x9/31.ts
I0424 16:30:13.637657 28004 s3.go:140] Saved to S3 https:/livepeer-cbs.s3.amazonaws.com/test1/987df0f8/P360p30fps16x9/31.ts
I0424 16:30:13.637681 28004 s3.go:131] Saving to S3 https:/livepeer-cbs.s3.amazonaws.com/test1/987df0f8/P240p30fps16x9/31.ts
E0424 16:30:19.647264 28004 s3.go:186] Post https://livepeer-cbs.s3.amazonaws.com: EOF
E0424 16:30:19.647289 28004 s3.go:135] Save S3 error: Post https://livepeer-cbs.s3.amazonaws.com: EOF
```

Logs from the B:

```
I0424 16:30:09.863100 28636 segment_rpc.go:287] Uploaded segment 31
E0424 16:30:17.785442 28636 segment_rpc.go:296] Unable to read response body for segment 31 : net/http: request canceled (Client.Timeout exceeded while reading body)
I0424 16:30:17.787038 28636 segment_rpc.go:264] Submitting segment 31 : 56 bytes
I0424 16:30:23.828139 28636 segment_rpc.go:287] Uploaded segment 31
I0424 16:30:24.844329 28636 segment_rpc.go:351] Successfully transcoded segment 31
E0424 16:30:26.108736 28636 lpms.go:174] Error polling Segment: open /Users/fribe0409/.lpData/offchain/19257e57_33.ts: no such file or directory, Retrying
I0424 16:30:26.110437 28636 segment_rpc.go:264] Submitting segment 32 : 56 bytes
I0424 16:30:26.202548 28636 segment_rpc.go:287] Uploaded segment 32
E0424 16:30:26.612331 28636 lpms.go:174] Error polling Segment: open /Users/fribe0409/.lpData/offchain/19257e57_33.ts: no such file or directory, Retrying
E0424 16:30:27.116633 28636 lpms.go:200] Error segmenting stream: open /Users/fribe0409/.lpData/offchain/19257e57_33.ts: no such file or directory
E0424 16:30:27.116768 28636 basic_rtmp_videostream.go:109] Error reading packet from RTMP: read tcp 127.0.0.1:1935->127.0.0.1:53763: use of closed network connection
I0424 16:30:27.116816 28636 mediaserver.go:301] Ended stream with id=test1
I0424 16:30:27.131346 28636 segment_rpc.go:351] Successfully transcoded segment 32
```

**To Reproduce**
Steps to reproduce the behavior (this is not the the original steps because we weren't sure why we got the `EOF` error from S3. The steps here try to simulate a S3 error).
1. Run a B/O/T local setup with S3 credentials
2. Run a video stream to B
3. Turn the S3 credentials to "inactive" in IAM
4. S3 saves should fail after a while. The broadcaster playlist will be "stuck". (interestingly, if you turn the S3 credential back on, the broadcaster playlist will continue!)

When we discovered this issue, the stream never recovered. (Unliked in the repro steps, the stream seems to recover after S3 credential is turned back on).

**Expected behavior**
Since the transcoding still succeeded, the broadcaster's playlist shouldn't be stuck, but instead, the playlist url should switch to the local url.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.