livepeer / livepeer/go-livepeer

On-disk local storage

Open
#867 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area: transcoding status: icebox type: feature
Dominant language
Go
Stars
586
Forks
226
Avg merge
1d 17h
Merged PRs (30d)
19

Description

**Is your feature request related to a problem? Please describe.**
Currently, segments within in-memory local storage are rotated out after about a [dozen segments](https://github.com/livepeer/go-livepeer/blob/master/drivers/local.go#L12).

For various reasons, an external OS may not be available or desirable (cost, latency, centralization, ease of testing or integration).

On-disk local storage would meet our persistence requirements without external OS. Some features which may drive this requirement:

* Recording/DVR or time shifting of live streams
* VOD playback
* Caching segments locally for verification
* Faster local accesses: https://github.com/livepeer/go-livepeer/issues/846#issuecomment-489236016
* External OS latency

**Describe the solution you'd like**

On-disk local storage driver. There are a few ways to integrate this:

* Replace in-memory storage entirely
* Keep this separate from in-memory storage and select which to use at start-up time via CLI flag
* Hybrid approach (serve from in-memory if in cache, otherwise hit disk). Questionable whether this is actually significantly faster given the page cache is usually pretty good. Needs benchmarking (and benchmarking might entail spending time implementing it; catch-22)
* Hybrid approach based on usage. Eg, VOD/DVR served from disk while live streams are served from memory.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with drivers/local.go and trace how in-memory segments are rotated and accessed. Compare the issue's replacement, separate CLI-selected, and hybrid approaches, noting that the hybrid option requires benchmarking. Done means the project has a defined on-disk local storage integration that provides the requested persistence for recording, VOD, caching, or related use cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.