Background video effects/virtual background improvements
Nobody has claimed this yet.
- Dominant language
- PHP
- Stars
- 2.2k
- Forks
- 587
- Avg merge
- 18h 27m
- Merged PRs (30d)
- 333
Description
How to use GitHub
- Please use the 👍 reaction to show that you are interested into the same feature.
- Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
- Subscribe to receive notifications on status change and new comments.
Feature request
Which Nextcloud Version are you currently using: (see administration page)
Nextcloud Hub 10 (31.0.9)
Talk 21.1.5
Is your feature request related to a problem? Please describe.
As of today, the video segmentation seems to be performed by the MediaPipe model on the Canvas 2D + CPU pipeline.
It works, but it is very CPU-intensive on less powerful hardware and the quality is definitely inferior to Google Meet/Zoom/etc.
I suppose other (newer/better) models also exist as of today, but I am no expert in the field.
Describe the solution you'd like
A higher quality and less CPU-intensive method for the virtual background functionality.
Describe alternatives you've considered
While looking around, I stumbled upon https://github.com/Volcomix/virtual-background, which has some additional implementations/models and discovered that the quality and performance can improve when using the WebGL pipeline or maybe the ML Kit model.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No implementation files or tests are named. Start by locating the current MediaPipe video-segmentation path and its Canvas 2D + CPU pipeline, then compare the linked virtual-background implementations and the WebGL or ML Kit options. Done would mean a higher-quality virtual background with lower CPU usage, but the issue does not define measurable targets.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- audio-video-rtc, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100