Linux: software encoder fails on GNOME portal/PipeWire BGRA (scale Invalid argument / imgutils ABRT) when no HW encode available

Nobody has claimed this yet.

### Is there an existing issue for this?

- [x] I have searched the existing issues

Related strings appear elsewhere but are **not the same setup**:

- #5309 / #2827 — `"Couldn't scale frame"` often with NVENC / other capture stacks
- This report is **forced `encoder = software` (libx264)** + **`capture = portal`** + **GNOME Wayland virtual monitor (Meta-0 / headless remote)** + GPU with **VAAPI decode only (no encode)**

### Is your issue described in the documentation?

- [ ] I have read the documentation

Docs cover portal capture and software encode; they do not document this failure path (portal BGRA → software scale fails, then fatal “no display/encoder” even though a monitor and PipeWire stream exist).

### Is your issue present in the latest beta/pre-release?

This issue is present in the latest pre-release

Tested:

- **Nightly `0.0.5368`** (`commit: 4eb2db5f4884135ef593ec375e003523b019800d`, debian-trixie-amd64)
- **Release `2026.516.143833`** (`commit: 14ffa6fdaa53f7b51512be2b3d24f3939695403c`)

### Describe the Bug

On **Debian 13 (trixie) + GNOME Wayland**, with a **virtual/remote monitor `Meta-0` 1920×1080** (GNOME Remote Desktop headless-style session), Sunshine is configured with:

- `capture = portal`
- `encoder = software` (required: host GPU has **no H.264 encode** entrypoint)

Portal capture succeeds and PipeWire delivers **format 12 (BGRA) @ 1920×1080**, `using memory buffers`.

Software encode then fails:

| Build | Behavior |
|--------|----------|
| **Nightly 0.0.5368** | During **encoder probe**: `Error: Couldn't scale frame: Invalid argument` → `Encoder [software] failed` → nvenc/vulkan/vaapi all fail → **`Fatal: Unable to find display or encoder during startup`** / `Video failed to find working encoder` |
| **2026.516.143833** | Encoder probe can pass; stream starts; **Frame 1 IDR** is produced; ~1s later process **aborts** on FFmpeg assert: `src_linesize … >= bytewidth failed at libavutil/imgutils.c:350` (`SIGABRT`). Client may see ~one frame then a dead session. |

Misleading outcome: logs clearly show **Wayland `Meta-0`**, successful **portal RemoteDesktop Start**, and a **PipeWire stream**, yet the final fatal message blames a missing display/encoder.

`system_tray = disabled` is set here so Qt tray does not take down the process; tray is unrelated to the scale failure.

### Expected Behavior

- Portal BGRA frames should convert/scale cleanly into `libx264`, **or**
- Fail with a **clear, specific** error (unsupported pixel format / stride / swscale), **without**:
- aborting on FFmpeg asserts, or
- claiming “no display” when Meta-0 + portal/PipeWire are already working

### Steps To Reproduce

1. Debian 13, GNOME Wayland session with virtual monitor (e.g. GNOME Remote Desktop), output **`Meta-0` 1920×1080**.
2. GPU with **no hardware H.264 encode** (VAAPI VLD/decode only), so software encode is required.
3. Sunshine config essentials:

```ini
encoder = software
capture = portal
hevc_mode = 0
av1_mode = 0
stream_audio = disabled
system_tray = disabled
min_log_level = debug
```

4. Ensure XDG portal RemoteDesktop restore token / screen-share permission works (SelectSources/Start succeed; PipeWire node appears).
5. Start Sunshine; observe encoder probe logs (nightly fails at probe). Optionally connect Moonlight Desktop stream on `2026.516` to hit the later `imgutils` abort.

### Host Operating System

Linux

### Operating System Version

Debian GNU/Linux 13 (trixie), kernel 6.12.95+deb13-amd64

### Architecture

amd64/x86_64

### Package

Linux - `.deb` (LizardByte debian-trixie-amd64 / packaged `2026.516.143833` and nightly `0.0.5368`)

### GPU Type

AMD

### GPU Model

AMD CAICOS PRO (Radeon HD 7450) — DRM `radeon`, Mesa Gallium

### GPU Driver/Mesa Version

Mesa 25.0.7-2+deb13u1 (LLVM 19.1.7); libva 1.22 / 2.22.0

### Capture Method

XDG Portal (PipeWire)

### Apps

```json
(default / Desktop stream; not app-specific)
```

### Log output

**Portal + PipeWire (common):**

```text
Screencasting with XDG portal
[wayland] Name: Meta-0
[wayland] Resolution: 1920x1080
[pipewire] Video format: 12
[pipewire] Size: 1920x1080
[pipewire] Framerate (from compositor): 0/1 (variable rate capture)
[pipewire] using memory buffers
```

**Nightly 0.0.5368 (software probe):**

```text
Trying encoder [software]
Creating encoder [libx264]
[libx264] profile Constrained Baseline, level 4.2, 4:2:0, 8-bit
Error: Couldn't scale frame: Invalid argument
Creating encoder [libx264]
Error: Couldn't scale frame: Invalid argument
Encoder [software] failed
Error: Couldn't find any working encoder matching [software]
… nvenc / vulkan / vaapi fail …
Fatal: Unable to find display or encoder during startup.
Fatal: Please check that a display is connected and powered on.
Error: Video failed to find working encoder
```

**2026.516.143833 (after stream start):**

```text
Creating encoder [libx264]
Frame 1: IDR Keyframe (AV_FRAME_FLAG_KEY)
New streaming session started [active sessions: 1]
Error: Assertion ((src_linesize) >= 0 ? (src_linesize) : (-(src_linesize))) >= bytewidth failed at libavutil/imgutils.c:350
# process SIGABRT (e.g. systemd status=6/ABRT)
```

**VAAPI (expected on this GPU — decode only):**

```text
vaapi vendor: Mesa Gallium driver … for AMD CAICOS …
[h264_vaapi] No usable encoding entrypoint found for profile VAProfileH264High
Could not open codec [h264_vaapi]: Function not implemented
Encoder [vaapi] failed
```

### Online logs

_No response_

### Additional Context

- Suspect **portal/PipeWire BGRA memory buffer layout (stride/planes)** vs FFmpeg sws / `av_image_copy` on the software path; `imgutils` assert matches **src_linesize < bytewidth** for the assumed format.
- Headless **Meta-*** virtual outputs + GNOME portal matter for remote/server-style hosts; software encode is the only path when VAAPI has no encode entrypoint.
- Happy to attach full debug logs, `vainfo`, and portal/PipeWire node details if useful.

### Related

- https://github.com/LizardByte/Sunshine/issues/5309 — same scale error string, different encoder/capture stack
- https://github.com/LizardByte/Sunshine/issues/2827 — same scale error string in other contexts

Start by tracing the portal/PipeWire BGRA memory-buffer path into the software libx264 encoder, comparing the encoder-probe failure with the later stream-start abort. Reproduce the Debian 13 GNOME Wayland setup using the listed configuration and logs. Done means BGRA frames encode successfully or produce a specific error without an FFmpeg assertion or misleading missing-display message.

Written by the indexing model from the issue text.