antirez / antirez/h3.c

RES audio sampling accumulates low-step error on the video sigma grid

Ouverte
#6 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
C
Étoiles
2.7k
Forks
210
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

## Problem

The public `h3_dit_denoise` RES path converts the audio velocity into a
video-sigma denoised estimate and advances it on `sigmas.video`:

```c
audio_denoised = audio_latent +
sigma_video * time_shift_slope(sigma_video) * audio_velocity;

h3_res_step(..., sigmas.video, ...);
```

The audio output is conditioned on the audio timestep, and the repository
already constructs a separate `sigmas.audio` schedule. Expressing the audio
trajectory through the video coordinate is valid in continuous time, but it
introduces substantial finite-step error in sparse RES sampling.

The Euler paths already advance audio directly on `sigmas.audio`.

## Why the discretizations differ

For the current video and audio shifts:

$$
\sigma_a=f(\sigma_v)=\frac{\sigma_v}{4-3\sigma_v}
$$

The existing reparameterized estimate is:

$$
D_v=x+\sigma_v f'(\sigma_v)v_a
$$

The native audio estimate is:

$$
D_a=x+\sigma_a v_a
$$

With constant audio velocity, $D_a$ is constant. $D_v$ remains nonlinear, so
a sparse multistep approximation has more truncation error.

```mermaid
flowchart LR
V["Audio velocity v_a
conditioned on sigma_a"]

V --> O["Video-coordinate estimate
D_v = x + sigma_v f'(sigma_v) v_a"]
O --> OV["RES on sigmas.video"]

V --> N["Audio-coordinate estimate
D_a = x + sigma_a v_a"]
N --> NA["RES on sigmas.audio"]
```

## Reproduction

Use a scalar sample initialized to `1.0` and a constant data-ward velocity of
`0.5`. Integrating from sigma 1 to 0 has the exact endpoint `1.5`.

| Steps | Existing formulation | Native audio formulation | Expected | Existing endpoint error |
|---:|---:|---:|---:|---:|
| 4 | 1.843430758 | 1.500000000 | 1.5 | 22.8954% |
| 7 | 1.620174885 | 1.500000000 | 1.5 | 8.0117% |
| 20 | 1.494925022 | 1.500000238 | 1.5 | 0.3383% |

At four steps, the video-coordinate result has a 22.8954% endpoint error. At
20 steps, the error falls to 0.3383%, making the behavior much less visible at
denser step counts.

These are deterministic solver-state measurements, not decoded-audio quality
measurements.

## Proposed outcome

Pair the velocity-to-denoised conversion with the sigma grid used by RES:

- Video: `video_latent + sigma_video * video_velocity`, then
`sigmas.video`.
- Audio: `audio_latent + sigma_audio * audio_velocity`, then
`sigmas.audio`.

Add a constant-velocity regression at sparse and ordinary step counts. The
Euler paths and schedule construction should remain unchanged.

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.