nextcloud / nextcloud/spreed

Allow external HW-encoded streams (AV1/VPx/H.264) to join Talk rooms

Open
#16,424 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement Needs triage
Dominant language
JavaScript
Stars
2.2k
Forks
586
Avg merge
18h 27m
Merged PRs (30d)
333

Description

Feature request

Which Nextcloud Version are you currently using:
32.0.2, NC Talk 22.0.4 I use the newest HPB as well.


Is your feature request related to a problem? Please describe.

Modern GPUs (Intel Arc, AMD RDNA3, etc.) already support hardware video encoding
including AV1, VP9, and H.264. However, browsers on Linux do not reliably expose
hardware encoding for WebRTC (via vaapi - especially encoding is not supported well on Linux). As a result, during Nextcloud Talk calls encoding
falls back to CPU, which makes high-quality video (e.g. 1080p or simulcast) very
CPU demanding even on powerful machines.

At the same time, using GStreamer, FFmpeg or OBS outside the browser, it is
already possible to generate real-time hardware-encoded AV1/VP9/H.264 streams (I am able to create perfect streams even for simulcast even in high resolutions).
Currently, there is no supported way to publish such an external WebRTC stream
into a Talk call.


Describe the solution you'd like

I would like Talk to support a mechanism or protocol that allows:

  1. A user to authenticate / join a Talk room, and
  2. Instead of using the browser’s built-in encoder, publish an external WebRTC
    stream (e.g. from GStreamer, FFmpeg, OBS or a WHIP-compatible client) into the
    same room.

This could be conceptually similar to WHIP/WHEP, but integrated with Talk’s
existing signaling and permissions, so the external encoder appears as a regular
participant in the call.


Describe alternatives you've considered

  • Using custom Chromium builds with hardware encoding: works only partially and is
    currently unreliable for WebRTC on Linux.
  • Joining Janus directly: not feasible because Talk rooms are dynamically created
    and not exposed to external publishers.
  • Software encoding simulcast: technically works but is too CPU-intensive for
    high-resolution or multi-layer streams.

None of these provide a clean, supported solution.


Additional context

Supporting external hardware-encoded streams would offer many benefits:

  • Enables high-quality AV1/VP9/H.264 publishing long before browsers expose HW
    encoding.
  • Dramatically reduces CPU load for users who need high-quality video.
  • Allows integrations with GStreamer pipelines, OBS, FFmpeg, and hardware-based
    encoding workflows.
  • Opens the door to additional features such as server-side recording bridges,
    monitoring tools, or experimenting with simulcast/SVC setups.

I would be happy to test this feature with my Intel Arc and AMD RDNA3 hardware on Linux. This is a perfect GPU combo, but my obstacle is the software (I can't use the perfect encoding/decoding codec feature of my Intel Arc GPU).

I'm willing to experiment if you show me the right direction.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files or tests are identified in the issue. Start by tracing Talk's existing signaling, authentication, permissions, and Janus integration, then compare them with the proposed WHIP-like external publishing flow. Done means an authenticated external encoder can join a Talk room and appear as a regular participant.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
audio-video-rtc
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.