paritytech / paritytech/host-rust-core

RFC: Host-mediated realtime media sessions

Open
#838 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement rfc
Dominant language
Swift
Stars
10
Forks
3
Avg merge
1d 10h
Merged PRs (30d)
148

Description

Source RFC: #839 · docs/rfcs/realtime-media-sessions.md
Core implementation: to follow

Summary

A Media service that lets a product run an audio or video call, with screen
sharing, without implementing, embedding, or observing any realtime transport.
The Host owns the connections, the capture devices, the codecs, the audio route,
and the pictures on screen. The product carries opaque signalling over a channel
it already has, says where each participant's picture belongs, and is told how
the call is going.

This is the tracking issue for the implementation. The first implementation is
WebRTC, with the Host running the peer connections. Nothing a product sees says
so: it handles sealed messages and Host-minted handles, not transport detail.

Requirements

  • No media reaches a product: no frames, no tracks, no device handles.
  • Signalling belongs to the Host: it offers, answers and exchanges candidates
    over the statement store, and a product neither carries nor sees any of it. No
    session detail and no participant's address reaches product code, and nothing
    reports bitrates, round-trip times, candidates, or addresses. The Host owns that subscription, so calling needs no
    statement permission of its own.
  • A call may be audio only. Camera and screen sharing are optional per
    participant and per direction.
  • A call needs no visible surface: audio connects while the product is still
    opening, so a background worker can answer.
  • The Host draws each incoming picture — a participant's camera or their shared
    screen — into rectangles the product places, at a depth the product chooses.
    Rectangles are the contract for every product; an element-based attach helper
    is SDK sugar over them.
  • Screen capture is not a device permission: the Host runs its own picker, the
    user chooses what is shared, and the product never names a window or a
    display.
  • Camera and microphone use the existing device permissions.
  • Devices belong to the Host: which microphone and camera are used, gain, echo
    cancellation, routing, and the in-call affordance for switching them. A
    product states what it wants sent and may express a preference, such as the
    front-facing camera or a speakerphone-style call; the Host or the OS may
    override it, and the product is told what is actually live and what kind of
    device the Host chose, so its call UI can show the truth. A product never
    learns a device name, model, or list.
  • Calling is its own permission, asked once and remembered, not the existing
    WebRTC grant whose load-time resolution is browser-specific. A Host may offer
    to allow a single call instead, which is prompt behaviour rather than a second
    kind of grant.
  • A remembered grant cannot become an invisible call: the Host shows an in-call
    indicator a product cannot suppress and the user can end the call from.
  • A call ends when the user withdraws camera or microphone access, or revokes
    the calling permission.
  • Participants are Host-minted handles. The Host is never told a product's own
    contact identities.
  • Whether a call connects directly or through a relay is the Host's choice and
    invisible to the product.
  • A Host that cannot provide the whole stack reports the service as unsupported
    rather than working partially.
  • Calls between more than two people are allowed. A Host must carry at least
    six participants, which needs no new server; beyond that it may refuse, and
    the refusal is reported so a product can grey out the control. Larger calls
    would need infrastructure this work does not cover.
  • Recording and product-chosen codecs are out of scope, as is waking a product
    that is not running for an incoming call.
  • The product keeps what it already owns: who may call whom, peer identity,
    ringing and decline, and call history.

Core implementation scope

  • Media service definition, versioned types, and wire ids.
  • Dispatcher and generated product clients.
  • Session lifecycle, participants joining and leaving, ownership, consent, and
    teardown on permission withdrawal.
  • Host-owned signalling over the statement store.
  • Host WebRTC binding: peer connections, capture including the screen picker,
    echo cancellation, audio session, connectivity with Host-minted relay
    credentials, and compositing.
  • Conformance fixtures for the privacy guarantees, not only a working call.
  • Reference product flow: invite, accept, end.

Implementation references

  • RFC document: #839
  • Background delivery to a product that is not running: handled separately

Target products

  • DIM2, as an SPA
  • Chat, as an SPA
  • T3ams, for audio and video calling with screen sharing
  • Meet, including screen sharing
  • any other product needing peer-to-peer audio, video, or sharing

Tasks

  • RFC document body — #839
  • RFC review and acceptance
  • Media service definition, versioned types, wire ids
  • Dispatcher and generated clients
  • Session lifecycle, ownership, consent
  • Host-owned signalling and incoming-call delivery
  • Compositing contract for cameras and shared screens, including clamping
    rules
  • Device ownership and live-state reporting contract
  • Calling permission, including revocation ending a live call
  • Privacy conformance fixtures
  • Product client wrappers
  • Reference flow: invite, accept, end
  • Decide where the WebRTC binding lives: a trait each Host implements, or
    shared Host code
  • Decide whether a background worker may start an audio-only call before
    the product's surface is open
  • Host adoption
    • Epoca
    • dotli-community
    • iOS
    • Android

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with docs/rfcs/realtime-media-sessions.md and the RFC review and acceptance task from #839. Map the listed service, dispatcher, lifecycle, signalling, WebRTC, privacy fixture, and reference-flow tasks, including the unresolved implementation choices. Done means the accepted contract is implemented across the listed scope and its privacy guarantees and invite, accept, and end flow are covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
swift
Domain
api, audio-video-rtc, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.