Allow external HW-encoded streams (AV1/VPx/H.264) to join Talk rooms
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 2.2k
- Forks
- 586
- Avg merge
- 18h 27m
- Merged PRs (30d)
- 333
Description
Feature request
Which Nextcloud Version are you currently using:
32.0.2, NC Talk 22.0.4 I use the newest HPB as well.
Is your feature request related to a problem? Please describe.
Modern GPUs (Intel Arc, AMD RDNA3, etc.) already support hardware video encoding
including AV1, VP9, and H.264. However, browsers on Linux do not reliably expose
hardware encoding for WebRTC (via vaapi - especially encoding is not supported well on Linux). As a result, during Nextcloud Talk calls encoding
falls back to CPU, which makes high-quality video (e.g. 1080p or simulcast) very
CPU demanding even on powerful machines.
At the same time, using GStreamer, FFmpeg or OBS outside the browser, it is
already possible to generate real-time hardware-encoded AV1/VP9/H.264 streams (I am able to create perfect streams even for simulcast even in high resolutions).
Currently, there is no supported way to publish such an external WebRTC stream
into a Talk call.
Describe the solution you'd like
I would like Talk to support a mechanism or protocol that allows:
- A user to authenticate / join a Talk room, and
- Instead of using the browser’s built-in encoder, publish an external WebRTC
stream (e.g. from GStreamer, FFmpeg, OBS or a WHIP-compatible client) into the
same room.
This could be conceptually similar to WHIP/WHEP, but integrated with Talk’s
existing signaling and permissions, so the external encoder appears as a regular
participant in the call.
Describe alternatives you've considered
- Using custom Chromium builds with hardware encoding: works only partially and is
currently unreliable for WebRTC on Linux. - Joining Janus directly: not feasible because Talk rooms are dynamically created
and not exposed to external publishers. - Software encoding simulcast: technically works but is too CPU-intensive for
high-resolution or multi-layer streams.
None of these provide a clean, supported solution.
Additional context
Supporting external hardware-encoded streams would offer many benefits:
- Enables high-quality AV1/VP9/H.264 publishing long before browsers expose HW
encoding. - Dramatically reduces CPU load for users who need high-quality video.
- Allows integrations with GStreamer pipelines, OBS, FFmpeg, and hardware-based
encoding workflows. - Opens the door to additional features such as server-side recording bridges,
monitoring tools, or experimenting with simulcast/SVC setups.
I would be happy to test this feature with my Intel Arc and AMD RDNA3 hardware on Linux. This is a perfect GPU combo, but my obstacle is the software (I can't use the perfect encoding/decoding codec feature of my Intel Arc GPU).
I'm willing to experiment if you show me the right direction.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are identified in the issue. Start by tracing Talk's existing signaling, authentication, permissions, and Janus integration, then compare them with the proposed WHIP-like external publishing flow. Done means an authenticated external encoder can join a Talk room and appear as a regular participant.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- audio-video-rtc
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100