Brooooooklyn / Brooooooklyn/webcodecs-node

VideoEncoder outputs EncodedVideoChunk in presentation order instead of decode order when B-frames are enabled

Open
#28 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
92
Forks
2
Avg merge
2d 4h
Merged PRs (30d)
1

Description

Hi, thank you for the awesome projects!
If my understanding is incorrect, please forgive me—I was working on a video export library using mediabunny and @napi-rs/webcodecs. When I encountered an issue where the frame number became 15324... when using the quality mode, I decided to investigate and discovered the following:

## Summary

When `latencyMode` is not set (defaulting to `'quality'` mode which enables B-frames), `VideoEncoder` outputs `EncodedVideoChunk` in **presentation order** instead of **decode order**. This violates the WebCodecs specification.

## WebCodecs Specification

According to the [WebCodecs spec](https://www.w3.org/TR/webcodecs/#dom-encodedvideochunk-timestamp):

> For EncodedVideoChunks the timestamp is the **decoding timestamp**... chunks **must** be returned in decode order.

The `timestamp` property of `EncodedVideoChunk` should be the **Decode Timestamp (DTS)**, not the Presentation Timestamp (PTS). When B-frames are present, these differ.

## Reproduction

```typescript
import { createCanvas } from '@napi-rs/canvas';
import {
Output,
Mp4OutputFormat,
FilePathTarget,
VideoSampleSource,
VideoSample,
QUALITY_HIGH,
} from 'mediabunny';

const WIDTH = 640;
const HEIGHT = 480;
const FPS = 30;
const TOTAL_FRAMES = 30;

async function main() {
console.log('=== Reproduction: B-frame timestamp issue ===\n');
console.log(`Encoding ${TOTAL_FRAMES} frames at ${FPS}fps with B-frames enabled.\n`);

const output = new Output({
format: new Mp4OutputFormat(),
target: new FilePathTarget('output-repro.mp4'),
});

const timestamps: number[] = [];
const videoSource = new VideoSampleSource({
codec: 'avc',
bitrate: QUALITY_HIGH,
onEncodedPacket: (packet) => {
timestamps.push(packet.timestamp);
},
});
output.addVideoTrack(videoSource, { frameRate: FPS });

await output.start();

const canvas = createCanvas(WIDTH, HEIGHT);
const ctx = canvas.getContext('2d');

for (let frame = 0; frame < TOTAL_FRAMES; frame++) {
// Draw frame number clearly visible
ctx.fillStyle = '#000';
ctx.fillRect(0, 0, WIDTH, HEIGHT);
ctx.fillStyle = '#fff';
ctx.font = 'bold 200px sans-serif';
ctx.textAlign = 'center';
ctx.textBaseline = 'middle';
ctx.fillText(String(frame), WIDTH / 2, HEIGHT / 2);

const imageData = ctx.getImageData(0, 0, WIDTH, HEIGHT);
const sample = new VideoSample(imageData.data.buffer, {
format: 'RGBA',
codedWidth: WIDTH,
codedHeight: HEIGHT,
timestamp: frame / FPS,
duration: 1 / FPS,
});

await videoSource.add(sample);
sample.close();
}

await output.finalize();

console.log('EncodedVideoChunk timestamps received (should be DTS, but are PTS):');
for (let i = 0; i < Math.min(10, timestamps.length); i++) {
console.log(` [${i}] ${timestamps[i].toFixed(6)}s`);
}

const isMonotonic = timestamps.every((t, i) => i === 0 || t >= timestamps[i - 1]);
console.log(`\nTimestamps monotonically increasing: ${isMonotonic}`);
console.log('(If true, timestamps are PTS not DTS - this is the bug)\n');

console.log('Output: output-repro.mp4');
console.log('\nVerify with ffprobe:');
console.log(' ffprobe -v error -show_frames -select_streams v:0 \\');
console.log(' -show_entries frame=pict_type,pts_time output-repro.mp4 | head -40');
console.log('\nExpected: PTS should be 0.000, 0.033, 0.066, 0.100, ...');
console.log('Actual: PTS is scrambled due to incorrect timestamp handling');
}

main().catch(console.error);
```

## Actual Result

```
Output order:
idx=0, timestamp=0ms, type=key
idx=1, timestamp=33ms, type=delta
idx=2, timestamp=67ms, type=delta
idx=3, timestamp=100ms, type=delta
...
Strictly increasing timestamps: true
```

The timestamps are strictly increasing, indicating **presentation order**.

## Expected Result

With B-frames (I, B, B, P pattern), the output should be in **decode order**, where timestamps would NOT be strictly increasing. For example:

```
idx=0, timestamp=0ms, type=key (I-frame)
idx=1, timestamp=100ms, type=delta (P-frame, decode first)
idx=2, timestamp=33ms, type=delta (B-frame, decode after P)
idx=3, timestamp=67ms, type=delta (B-frame, decode after P)
...
```

## Impact

This causes issues when muxing to MP4, because muxers (like mediabunny) expect chunks in decode order per the WebCodecs spec. The resulting video has incorrect playback order - frames appear out of sequence (e.g., 0, 4, 2, 1, 3... instead of 0, 1, 2, 3, 4...).

## Analysis

Looking at `src/webcodecs/video_encoder.rs`:

1. **B-frames are enabled** (lines ~1567-1570):
```rust
if latency_mode == LatencyMode::Quality {
encoder_ctx.set_max_b_frames(2);
}
```

2. **Output loop** (lines ~1172-1316): Packets from FFmpeg are passed directly to the output callback without sorting by DTS.

FFmpeg returns packets in the order they're produced, which for B-frames is presentation order. The fix would be to buffer packets and sort them by DTS before calling the output callback, or to use `chunk.timestamp = dts` instead of `chunk.timestamp = pts`.

## Suggested Fix

In the encoding output loop, either:
1. Set `EncodedVideoChunk.timestamp` to the packet's DTS (not PTS)
2. Or buffer packets and emit them sorted by DTS

## Environment

- @napi-rs/webcodecs: latest
- OS: macOS
- Node.js: v22

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.