livepeer / livepeer/catalyst

IPFS support

Open
#154 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
24
Forks
16
Avg merge
1d 8h
Merged PRs (30d)
1

Description

## General considerations
- We are not willing to run our own IPFS nodes at this point, therefore, we need to upload (pin) the files to cloud provider, like [Pinata],(https://www.pinata.cloud/) through their proprietary API. To ensure the cloud provider didn't tamper with the contents of the file, we probably want to add an extra step of calculating the content hash locally and comparing it with the address, returned after upload.
- Files could be accessed by HTTP through [public](https://ipfs.github.io/public-gateway-checker/) or [private](https://docs.pinata.cloud/gateways/dedicated-gateways) gateway providers. To operate in a trustless manner, the content hash should be calculated and matched against the returned CID. The minimal unit, which can be validated, is an entire file. An [example](https://ipfs.io/ipfs/Qme7ss3ARVgxv6rXqVPiikMJ8u2NLgmgszg13pYrDKEoiu) of IPFS URL, which uses public gateway.
- With content-based addressing, the target URL is not input, but output parameter, because it includes content hash. It may require a slightly different flow, compared to S3. It also may present several challenges with playlist generation e.g. a segment should be fully uploaded, before playlist entry can be added; playlist URL will change every time it's updated. [IPNS](https://docs.ipfs.tech/concepts/ipns/) and [DNSLink](https://docs.ipfs.tech/concepts/dnslink/) are mechanisms which could be used to address that.
## VOD
### Input
- IPFS URLs do not explicitly contain file type. Most widely used IPFS gateway implementation automatically sets `Content-type` header in HTTP GET response, we can use that, but it's [not a mandatory requirement for gateway implementations](https://github.com/ipfs/specs/blob/main/http-gateways/PATH_GATEWAY.md#content-type-response-header).
### Output
- `ipfs://` URL support should be added to [go-tools](https://github.com/livepeer/go-tools/issues/3) and `catalyst-uploader`. It will use Pinata API to pin the file and return Pinata IPFS gateway URL. Content hash matching should be implemented as well.
- when segment is uploaded, Mist needs to use URL returned from `catalyst-uploader` in the playlist
- TBD: what should be the playlist behavior? It seems that IPNS is not available on Pinata. It seem to support [directory wrapping](https://flyingzumwalt.gitbooks.io/decentralized-web-primer/content/files-on-ipfs/lessons/wrap-directories-around-content.html), with pretty obscure API, which may allow to address the file by name.
## Live
- The main issue is dynamic playlist files
- TBD

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the referenced go-tools issue and the catalyst-uploader entry point, then trace how Mist currently handles S3 uploads and playlist updates. The Pinata API, local content-hash validation, returned gateway URLs, and live playlist behavior all need decisions; done is not defined until the VOD and Live flows are specified.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cloud, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.