selkies-project / selkies-project/selkies

[META] Windows/MacOS support mega thread

Open
#57 26 comments 4 reactions 0 assignees View on GitHub
encoding enhancement interface
Dominant language
Python
Stars
2.2k
Forks
193
Avg merge
20h 54m
Merged PRs (30d)
36

Description

**NOTE: We will not accept a PR regarding this issue right now, due to the volume of code it modifies, unless there is a dedicated and determined maintainer to continue maintaining these components. There are free (but not necessarily open source) alternatives in these OSes, including Reemo and Parsec, for now.**

**NOTE: This needs a maintainer dedicated to each or both OS. Current maintainers are focused on Linux usage.**

---

Can be AI-driven: Yes, as long as all cases are tested carefully in real-life by people as well as automated tests. However, **expert guidance is required due to working with OS intrinsics and media pipelines**.
Expected to be workable with: Fable 5, GPT-5.6 Sol, or equivalent (max thinking, requires expert guidance, especially with Opus 5, Kimi-K3, or under).

Update: https://crates.io/crates/screencapturekit (and avf as fallback) should be used for macOS. For Windows, we aim to support the latest versions of Windows 11 (d3d12), with another API that supports later Windows 10 versions (d3d11/mf; the two of these are actually hardware compatibility as far as I'm aware) as a fallback. The best API for this case, which uses zero-copy pipelines, should be used. For audio, macOS either uses the API in screencapturekit or CoreAudio; Windows uses the Windows Audio Session API available with Windows 10. If there is any other API that is better, need to use it.

---

Windows support is in progress with the contribution from the community.

Note that the below are based on GStreamer conventions and we now use pixelflux and pcmflux.

For Windows, the pipeline should add `d3d11screencapturesrc` (screen capture), `d3d11colorconvert` (color convert), and `wasapi` or `wasapi2` (audio capture).

In addition, it would be worth adding the `mfh264enc` and `mfvp9enc` encoder pipelines for Windows native encoding capabilities, as well as zero-copy `nvd3d11h264enc`, `nvd3d11h265enc`, and `nvd3d11av1enc` for NVIDIA, which also may be used for Windows.

For MacOS, `pyobjc-framework-applicationservices` and `pyobjc-framework-quartz` are possibly frequently used interfaces for input.

ScreenCaptureKit is preferred with `avfvideosrc capture-screen=true` as fallback(?), `glcolorconvert` (color convert, not sure), and `osxaudiosrc` (audio capture) are probably the way to go for media capture.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the current Linux capture, audio, and encoding pipeline alongside the pixelflux and pcmflux conventions mentioned here. Then assess the listed Windows and macOS APIs, including ScreenCaptureKit, d3d11screencapturesrc, WASAPI, and native encoders. Done requires expert-reviewed implementation plus automated and real-world testing on the supported operating systems.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
audio-video-rtc, desktop, operating-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.