Brooooooklyn / Brooooooklyn/webcodecs-node
VideoEncoder outputs EncodedVideoChunk in presentation order instead of decode order when B-frames are enabled
- Dominant language
- Rust
- Stars
- 92
- Forks
- 2
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 1
Description
Hi, thank you for the awesome projects!
If my understanding is incorrect, please forgive me—I was working on a video export library using mediabunny and @napi-rs/webcodecs. When I encountered an issue where the frame number became 15324... when using the quality mode, I decided to investigate and discovered the following:
## Summary
When `latencyMode` is not set (defaulting to `'quality'` mode which enables B-frames), `VideoEncoder` outputs `EncodedVideoChunk` in **presentation order** instead of **decode order**. This violates the WebCodecs specification.
## WebCodecs Specification
According to the [WebCodecs spec](https://www.w3.org/TR/webcodecs/#dom-encodedvideochunk-timestamp):
> For EncodedVideoChunks the timestamp is the **decoding timestamp**... chunks **must** be returned in decode order.
The `timestamp` property of `EncodedVideoChunk` should be the **Decode Timestamp (DTS)**, not the Presentation Timestamp (PTS). When B-frames are present, these differ.
## Reproduction
```typescript
import { createCanvas } from '@napi-rs/canvas';
import {
Output,
Mp4OutputFormat,
FilePathTarget,
VideoSampleSource,
VideoSample,
QUALITY_HIGH,
} from 'mediabunny';
const WIDTH = 640;
const HEIGHT = 480;
const FPS = 30;
const TOTAL_FRAMES = 30;
async function main() {
console.log('=== Reproduction: B-frame timestamp issue ===\n');
console.log(`Encoding ${TOTAL_FRAMES} frames at ${FPS}fps with B-frames enabled.\n`);
const output = new Output({
format: new Mp4OutputFormat(),
target: new FilePathTarget('output-repro.mp4'),
});
const timestamps: number[] = [];
const videoSource = new VideoSampleSource({
codec: 'avc',
bitrate: QUALITY_HIGH,
onEncodedPacket: (packet) => {
timestamps.push(packet.timestamp);
},
});
output.addVideoTrack(videoSource, { frameRate: FPS });
await output.start();
const canvas = createCanvas(WIDTH, HEIGHT);
const ctx = canvas.getContext('2d');
for (let frame = 0; frame < TOTAL_FRAMES; frame++) {
// Draw frame number clearly visible
ctx.fillStyle = '#000';
ctx.fillRect(0, 0, WIDTH, HEIGHT);
ctx.fillStyle = '#fff';
ctx.font = 'bold 200px sans-serif';
ctx.textAlign = 'center';
ctx.textBaseline = 'middle';
ctx.fillText(String(frame), WIDTH / 2, HEIGHT / 2);
const imageData = ctx.getImageData(0, 0, WIDTH, HEIGHT);
const sample = new VideoSample(imageData.data.buffer, {
format: 'RGBA',
codedWidth: WIDTH,
codedHeight: HEIGHT,
timestamp: frame / FPS,
duration: 1 / FPS,
});
await videoSource.add(sample);
sample.close();
}
await output.finalize();
console.log('EncodedVideoChunk timestamps received (should be DTS, but are PTS):');
for (let i = 0; i < Math.min(10, timestamps.length); i++) {
console.log(` [${i}] ${timestamps[i].toFixed(6)}s`);
}
const isMonotonic = timestamps.every((t, i) => i === 0 || t >= timestamps[i - 1]);
console.log(`\nTimestamps monotonically increasing: ${isMonotonic}`);
console.log('(If true, timestamps are PTS not DTS - this is the bug)\n');
console.log('Output: output-repro.mp4');
console.log('\nVerify with ffprobe:');
console.log(' ffprobe -v error -show_frames -select_streams v:0 \\');
console.log(' -show_entries frame=pict_type,pts_time output-repro.mp4 | head -40');
console.log('\nExpected: PTS should be 0.000, 0.033, 0.066, 0.100, ...');
console.log('Actual: PTS is scrambled due to incorrect timestamp handling');
}
main().catch(console.error);
```
## Actual Result
```
Output order:
idx=0, timestamp=0ms, type=key
idx=1, timestamp=33ms, type=delta
idx=2, timestamp=67ms, type=delta
idx=3, timestamp=100ms, type=delta
...
Strictly increasing timestamps: true
```
The timestamps are strictly increasing, indicating **presentation order**.
## Expected Result
With B-frames (I, B, B, P pattern), the output should be in **decode order**, where timestamps would NOT be strictly increasing. For example:
```
idx=0, timestamp=0ms, type=key (I-frame)
idx=1, timestamp=100ms, type=delta (P-frame, decode first)
idx=2, timestamp=33ms, type=delta (B-frame, decode after P)
idx=3, timestamp=67ms, type=delta (B-frame, decode after P)
...
```
## Impact
This causes issues when muxing to MP4, because muxers (like mediabunny) expect chunks in decode order per the WebCodecs spec. The resulting video has incorrect playback order - frames appear out of sequence (e.g., 0, 4, 2, 1, 3... instead of 0, 1, 2, 3, 4...).
## Analysis
Looking at `src/webcodecs/video_encoder.rs`:
1. **B-frames are enabled** (lines ~1567-1570):
```rust
if latency_mode == LatencyMode::Quality {
encoder_ctx.set_max_b_frames(2);
}
```
2. **Output loop** (lines ~1172-1316): Packets from FFmpeg are passed directly to the output callback without sorting by DTS.
FFmpeg returns packets in the order they're produced, which for B-frames is presentation order. The fix would be to buffer packets and sort them by DTS before calling the output callback, or to use `chunk.timestamp = dts` instead of `chunk.timestamp = pts`.
## Suggested Fix
In the encoding output loop, either:
1. Set `EncodedVideoChunk.timestamp` to the packet's DTS (not PTS)
2. Or buffer packets and emit them sorted by DTS
## Environment
- @napi-rs/webcodecs: latest
- OS: macOS
- Node.js: v22
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.