nextcloud / nextcloud/spreed

Background video effects/virtual background improvements

Open
#16,008 1 comment 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement feature: call 📹 feature: WebRTC 🚡
Dominant language
PHP
Stars
2.2k
Forks
587
Avg merge
18h 27m
Merged PRs (30d)
333

Description

How to use GitHub
  • Please use the 👍 reaction to show that you are interested into the same feature.
  • Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
  • Subscribe to receive notifications on status change and new comments.

Feature request

Which Nextcloud Version are you currently using: (see administration page)
Nextcloud Hub 10 (31.0.9)
Talk 21.1.5

Is your feature request related to a problem? Please describe.
As of today, the video segmentation seems to be performed by the MediaPipe model on the Canvas 2D + CPU pipeline.

It works, but it is very CPU-intensive on less powerful hardware and the quality is definitely inferior to Google Meet/Zoom/etc.

I suppose other (newer/better) models also exist as of today, but I am no expert in the field.

Describe the solution you'd like
A higher quality and less CPU-intensive method for the virtual background functionality.

Describe alternatives you've considered
While looking around, I stumbled upon https://github.com/Volcomix/virtual-background, which has some additional implementations/models and discovered that the quality and performance can improve when using the WebGL pipeline or maybe the ML Kit model.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No implementation files or tests are named. Start by locating the current MediaPipe video-segmentation path and its Canvas 2D + CPU pipeline, then compare the linked virtual-background implementations and the WebGL or ML Kit options. Done would mean a higher-quality virtual background with lower CPU usage, but the issue does not define measurable targets.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
audio-video-rtc, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.