meta-pytorch / meta-pytorch/torchcodec
Flaky JSON error in encoder tests
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 125
- Avg merge
- 22h 47m
- Merged PRs (30d)
- 54
Description
Context:
We have encoder tests that rely on ffprobe to compare our encoded media's metadata with ffmpeg's.
TestAudioEncoder.test_against_cliusesvalidate_frames_propertiesTestVideoEncoder.test_video_encoder_against_ffmpeg_cliuses_get_frames_info- These functions use
-of jsonto return the metadata as a JSON.
Problem:
These tests are flaky and sometimes fail on Windows + FFmpeg8 with the error: json.decoder.JSONDecodeError: Expecting ',' delimiter.
These tests normally pass after a retry, but currently they add noisy failures to PRs, so we should find and resolve the underlying cause, or switch to a more reliable output format such as ffprobe's default format, which we use in _get_video_metadata.
Example failures:
TestAudioEncoder::test_against_cli on Windows + FFmpeg8, for mp3 and flac:
- Observed here: https://github.com/meta-pytorch/torchcodec/actions/runs/24197621221/job/70640272981?pr=1326
================================== FAILURES ===================================
__ TestAudioEncoder.test_against_cli[to_file-flac-32000-1-999999999-asset0] ___
self = <test.test_encoders.TestAudioEncoder object at 0x000001D7D3AB19A0>
asset = TestAudio(filename='nasa_13013.mp4.audio.mp3', default_stream_index=0, stream_infos={0: TestAudioStreamInfo(sample_rat..., 2.3536e-04, 2.7501e-04, 2.1080e-04,
-2.1618e-05, -8.9567e-05, -4.4332e-04, -5.0099e-04, -2.7716e-04]])]})
bit_rate = 999999999, num_channels = 1, sample_rate = 32000, format = 'flac'
method = 'to_file'
tmp_path = WindowsPath('C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/test_against_cli_to_file_flac_38')
capfd = <_pytest.capture.CaptureFixture object at 0x000001D7D599EAD0>
with_ffmpeg_debug_logs = None
@needs_ffmpeg_cli
@pytest.mark.parametrize("asset", (NASA_AUDIO_MP3, SINE_MONO_S32))
@pytest.mark.parametrize("bit_rate", (None, 0, 44_100, 999_999_999))
@pytest.mark.parametrize("num_channels", (None, 1, 2))
@pytest.mark.parametrize("sample_rate", (8_000, 32_000))
@pytest.mark.parametrize(
"format",
[
# TODO: https://github.com/pytorch/torchcodec/issues/837
pytest.param(
"mp3",
marks=pytest.mark.skipif(
IS_WINDOWS and ffmpeg_major_version <= 5,
reason="Encoding mp3 on Windows is weirdly buggy",
),
),
pytest.param(
"wav",
marks=pytest.mark.skipif(
ffmpeg_major_version == 4,
reason="Swresample with FFmpeg 4 doesn't work on wav files",
),
),
"flac",
],
)
@pytest.mark.parametrize("method", ("to_file", "to_tensor", "to_file_like"))
def test_against_cli(
self,
asset,
bit_rate,
num_channels,
sample_rate,
format,
method,
tmp_path,
capfd,
with_ffmpeg_debug_logs,
):
# Encodes samples with our encoder and with the FFmpeg CLI, and checks
# that both decoded outputs are equal
encoded_by_ffmpeg = tmp_path / f"ffmpeg_output.{format}"
subprocess.run(
["ffmpeg", "-i", str(asset.path)]
+ (["-b:a", f"{bit_rate}"] if bit_rate is not None else [])
+ (["-ac", f"{num_channels}"] if num_channels is not None else [])
+ ["-ar", f"{sample_rate}"]
+ [
str(encoded_by_ffmpeg),
],
capture_output=True,
check=True,
)
encoder = AudioEncoder(self.decode(asset).data, sample_rate=asset.sample_rate)
params = dict(
bit_rate=bit_rate, num_channels=num_channels, sample_rate=sample_rate
)
if method == "to_file":
encoded_by_us = tmp_path / f"output.{format}"
encoder.to_file(dest=str(encoded_by_us), **params)
elif method == "to_tensor":
encoded_by_us = encoder.to_tensor(format=format, **params)
elif method == "to_file_like":
file_like = io.BytesIO()
encoder.to_file_like(file_like, format=format, **params)
encoded_by_us = file_like.getvalue()
else:
raise ValueError(f"Unknown method: {method}")
captured = capfd.readouterr()
if format == "wav":
assert "Timestamps are unset in a packet" not in captured.err
if format == "mp3":
assert "Queue input is backward in time" not in captured.err
if format in ("flac", "wav"):
assert "Encoder did not produce proper pts" not in captured.err
if format in ("flac", "mp3"):
assert "Application provided invalid" not in captured.err
assert_close = torch.testing.assert_close
if sample_rate != asset.sample_rate:
if platform.machine().lower() == "aarch64":
rtol, atol = 0, 1e-2
else:
rtol, atol = 0, 1e-3
if sys.platform == "darwin":
assert_close = partial(assert_tensor_close_on_at_least, percentage=99)
elif format == "wav":
rtol, atol = 0, 1e-4
elif format == "mp3" and asset is SINE_MONO_S32 and num_channels == 2:
# Not sure why, this one needs slightly higher tol. With default
# tolerances, the check fails on ~1% of the samples, so that's
# probably fine. It might be that the FFmpeg CLI doesn't rely on
# libswresample for converting channels?
rtol, atol = 0, 1e-3
else:
rtol, atol = None, None
if IS_WINDOWS_WITH_FFMPEG_LE_70 and format == "mp3":
# We're getting a "Could not open input file" on Windows mp3 files when decoding.
# TODO: https://github.com/pytorch/torchcodec/issues/837
return
samples_by_us = self.decode(encoded_by_us)
samples_by_ffmpeg = self.decode(encoded_by_ffmpeg)
assert_close(
samples_by_us.data,
samples_by_ffmpeg.data,
rtol=rtol,
atol=atol,
)
assert samples_by_us.pts_seconds == samples_by_ffmpeg.pts_seconds
assert samples_by_us.duration_seconds == samples_by_ffmpeg.duration_seconds
assert samples_by_us.sample_rate == samples_by_ffmpeg.sample_rate
if method == "to_file":
> validate_frames_properties(actual=encoded_by_us, expected=encoded_by_ffmpeg)
test\test_encoders.py:387:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
test\test_encoders.py:64: in validate_frames_properties
frames_actual, frames_expected = (
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
test\test_encoders.py:65: in <genexpr>
json.loads(
C:\Users\runneradmin\miniconda3\envs\test\Lib\json\__init__.py:352: in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
C:\Users\runneradmin\miniconda3\envs\test\Lib\json\decoder.py:345: in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <json.decoder.JSONDecoder object at 0x000001D7B88BF8C0>
s = '{\n "frames": [\n {\n "pts": 0,\n "pts_time": "0.000000",\n "duration": 23...": "0.024000",\n "sample_fmt": "s32",\n "nb_samples": 768,\n "channels": 1\n }'
idx = 0
def raw_decode(self, s, idx=0):
"""Decode a JSON document from ``s`` (a ``str`` beginning with
a JSON document) and return a 2-tuple of the Python
representation and the index in ``s`` where the document ended.
This can be used to decode a JSON document from a string that may
have extraneous data at the end.
"""
try:
> obj, end = self.scan_once(s, idx)
^^^^^^^^^^^^^^^^^^^^^^
E json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1640 column 10 (char 44777)
C:\Users\runneradmin\miniconda3\envs\test\Lib\json\decoder.py:361: JSONDecodeError
---------------------------- Captured stderr call -----------------------------
[AVFormatContext @ 000001D7CFE9D4C0] Opening 'C:\Users\runneradmin\AppData\Local\Temp\pytest-of-runneradmin\pytest-0\test_against_cli_to_file_flac_38\output.flac' for reading
[file @ 000001D7D30FBB00] Setting default whitelist 'file,crypto,data'
[flac @ 000001D7CFE9D4C0] Format flac probed with size=2048 and score=100
[flac @ 000001D7CFE9D4C0] Before avformat_find_stream_info() pos: 8286 bytes read:32768 seeks:0 nb_streams:1
[flac @ 000001D7CFE9D4C0] All info found
[flac @ 000001D7CFE9D4C0] After avformat_find_stream_info() pos: 28766 bytes read:32768 seeks:0 frames:1
[SWR @ 000001D7D338CA80] Using fltp internally between filters
[flac @ 000001D7CFE9D4C0] first_dts 0 not matching first dts 396288 (pts 396288, duration 2304) in the queue
[AVIOContext @ 000001D7CB58BB40] Statistics: 489375 bytes read, 0 seeks
[AVFormatContext @ 000001D7CFE9DCC0] Opening 'C:\Users\runneradmin\AppData\Local\Temp\pytest-of-runneradmin\pytest-0\test_against_cli_to_file_flac_38\ffmpeg_output.flac' for reading
[file @ 000001D7D30FC340] Setting default whitelist 'file,crypto,data'
[flac @ 000001D7CFE9DCC0] Format flac probed with size=2048 and score=100
[flac @ 000001D7CFE9DCC0] Before avformat_find_stream_info() pos: 8365 bytes read:32768 seeks:0 nb_streams:1
[flac @ 000001D7CFE9DCC0] All info found
[flac @ 000001D7CFE9DCC0] After avformat_find_stream_info() pos: 28845 bytes read:32768 seeks:0 frames:1
[SWR @ 000001D7D338CA80] Using fltp internally between filters
[flac @ 000001D7CFE9DCC0] first_dts 0 not matching first dts 396288 (pts 396288, duration 2304) in the queue
[AVIOContext @ 000001D7CB58F7C0] Statistics: 489454 bytes read, 0 seeks
=========================== short test summary info ===========================
FAILED test/test_encoders.py::TestAudioEncoder::test_against_cli[to_file-flac-32000-1-999999999-asset0] - json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1640 column 10 (char 44777)
=========== 1 failed, 1232 passed, 595 skipped in 547.98s (0:09:07) ===========
TestVideoEncoder::test_video_encoder_against_ffmpeg_cli on Windows + FFmpeg8 for flv, probably others.
- Observed here: https://github.com/meta-pytorch/torchcodec/actions/runs/24336485303/job/71057725479?pr=1328
================================== FAILURES ===================================
_ TestVideoEncoder.test_video_encoder_against_ffmpeg_cli[30-to_file-encode_params2-flv] _
self = <test.test_encoders.TestVideoEncoder object at 0x000001D382E05950>
tmp_path = WindowsPath('C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/test_video_encoder_against_ffm14')
format = 'flv'
encode_params = {'crf': None, 'pixel_format': 'yuv420p', 'preset': 'ultrafast'}
method = 'to_file', frame_rate = 30
@needs_ffmpeg_cli
@pytest.mark.parametrize(
"format",
(
"mov",
"mp4",
"avi",
"mkv",
"flv",
pytest.param(
"webm",
marks=[
pytest.mark.slow,
pytest.mark.skipif(
ffmpeg_major_version == 4
or (IS_WINDOWS and ffmpeg_major_version >= 6),
reason="Codec for webm is not available in this FFmpeg installation.",
),
],
),
),
)
@pytest.mark.parametrize(
"encode_params",
[
{"pixel_format": "yuv444p", "crf": 0, "preset": None},
{"pixel_format": "yuv420p", "crf": 30, "preset": None},
{"pixel_format": "yuv420p", "crf": None, "preset": "ultrafast"},
{"pixel_format": "yuv420p", "crf": None, "preset": None},
],
)
@pytest.mark.parametrize("method", ("to_file", "to_tensor", "to_file_like"))
@pytest.mark.parametrize("frame_rate", [30, 29.97])
def test_video_encoder_against_ffmpeg_cli(
self, tmp_path, format, encode_params, method, frame_rate
):
pixel_format = encode_params["pixel_format"]
crf = encode_params["crf"]
preset = encode_params["preset"]
if format in ("avi", "flv") and pixel_format == "yuv444p":
pytest.skip(f"Default codec for {format} does not support {pixel_format}")
source_frames = self.decode(TEST_SRC_2_720P.path)
# Encode with FFmpeg CLI
temp_raw_path = str(tmp_path / "temp_input.raw")
with open(temp_raw_path, "wb") as f:
f.write(source_frames.permute(0, 2, 3, 1).cpu().numpy().tobytes())
ffmpeg_encoded_path = str(tmp_path / f"ffmpeg_output.{format}")
# Some codecs (ex. MPEG4) do not support CRF or preset.
# Flags not supported by the selected codec will be ignored.
ffmpeg_cmd = [
"ffmpeg",
"-y",
"-f",
"rawvideo",
"-pix_fmt",
"rgb24", # Input format
"-s",
f"{source_frames.shape[3]}x{source_frames.shape[2]}",
"-r",
str(frame_rate),
"-i",
temp_raw_path,
]
if pixel_format is not None: # Output format
ffmpeg_cmd.extend(["-pix_fmt", pixel_format])
if preset is not None:
ffmpeg_cmd.extend(["-preset", preset])
if crf is not None:
ffmpeg_cmd.extend(["-crf", str(crf)])
# Output path must be last
ffmpeg_cmd.append(ffmpeg_encoded_path)
subprocess.run(ffmpeg_cmd, check=True)
ffmpeg_frames = self.decode(ffmpeg_encoded_path).data
# Encode with our video encoder
encoder = VideoEncoder(frames=source_frames, frame_rate=frame_rate)
encoder_output_path = str(tmp_path / f"encoder_output.{format}")
if method == "to_file":
encoder.to_file(
dest=encoder_output_path,
pixel_format=pixel_format,
crf=crf,
preset=preset,
)
encoder_frames = self.decode(encoder_output_path)
elif method == "to_tensor":
encoded_output = encoder.to_tensor(
format=format,
pixel_format=pixel_format,
crf=crf,
preset=preset,
)
encoder_frames = self.decode(encoded_output)
elif method == "to_file_like":
file_like = io.BytesIO()
encoder.to_file_like(
file_like=file_like,
format=format,
pixel_format=pixel_format,
crf=crf,
preset=preset,
)
encoder_frames = self.decode(file_like.getvalue())
else:
raise ValueError(f"Unknown method: {method}")
assert ffmpeg_frames.shape[0] == encoder_frames.shape[0]
# MPEG codec used for avi format does not accept CRF
percentage = 94 if format == "avi" else 99
# Check that PSNR between both encoded versions is high
for ff_frame, enc_frame in zip(ffmpeg_frames, encoder_frames):
res = psnr(ff_frame, enc_frame)
assert res > 30
assert_tensor_close_on_at_least(
ff_frame, enc_frame, percentage=percentage, atol=2
)
# Only compare video metadata on ffmpeg versions >= 6, as older versions
# are often missing metadata
if ffmpeg_major_version >= 6 and method == "to_file":
fields = [
"duration",
"duration_ts",
"r_frame_rate",
"time_base",
"nb_frames",
]
ffmpeg_metadata = self._get_video_metadata(
ffmpeg_encoded_path,
fields=fields,
)
encoder_metadata = self._get_video_metadata(
encoder_output_path,
fields=fields,
)
assert ffmpeg_metadata == encoder_metadata
# Check that frame timestamps and duration are the same
fields = ("pts", "pts_time")
if format != "flv":
fields += ("duration", "duration_time")
> ffmpeg_frames_info = self._get_frames_info(
ffmpeg_encoded_path, fields=fields
)
test\test_encoders.py:1172:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
test\test_encoders.py:669: in _get_frames_info
frames = json.loads(result.stdout)["frames"]
^^^^^^^^^^^^^^^^^^^^^^^^^
C:\Users\runneradmin\miniconda3\envs\test\Lib\json\__init__.py:352: in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
C:\Users\runneradmin\miniconda3\envs\test\Lib\json\decoder.py:345: in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <json.decoder.JSONDecoder object at 0x000001D3E8A36E40>
s = '{\n "frames": [\n {\n "pts": 0,\n "pts_time": "0.000000"\n },\n {\n ... "pts_time": "0.933000"\n },\n {\n "pts": 967,\n "pts_time": "0.967000"\n }'
idx = 0
def raw_decode(self, s, idx=0):
"""Decode a JSON document from ``s`` (a ``str`` beginning with
a JSON document) and return a 2-tuple of the Python
representation and the index in ``s`` where the document ended.
This can be used to decode a JSON document from a string that may
have extraneous data at the end.
"""
try:
> obj, end = self.scan_once(s, idx)
^^^^^^^^^^^^^^^^^^^^^^
E json.decoder.JSONDecodeError: Expecting ',' delimiter: line 122 column 10 (char 2412)
C:\Users\runneradmin\miniconda3\envs\test\Lib\json\decoder.py:361: JSONDecodeError
---------------------------- Captured stderr call -----------------------------
ffmpeg version 8.0.1 Copyright (c) 2000-2025 the FFmpeg developers
built with clang version 22.1.0
configuration: --prefix=/d/bld/ffmpeg_1773007679189/_h_env/Library --cc=clang.exe --cxx=clang++.exe --nm=llvm-nm --ar=llvm-ar --disable-doc --enable-openssl --enable-demuxer=dash --enable-hardcoded-tables --enable-libfreetype --enable-libharfbuzz --enable-libfontconfig --enable-libopenh264 --enable-libdav1d --ld=lld-link --target-os=win64 --enable-cross-compile --toolchain=msvc --host-cc=clang.exe --extra-libs=ucrt.lib --extra-libs=vcruntime.lib --extra-libs=oldnames.lib --strip=llvm-strip --disable-stripping --host-extralibs= --disable-libopenvino --enable-gpl --enable-libx264 --enable-libx265 --enable-libmp3lame --enable-libaom --enable-libsvtav1 --enable-libxml2 --enable-pic --enable-shared --disable-static --enable-version3 --enable-zlib --enable-libvorbis --enable-libopus --enable-librsvg --enable-libjxl --enable-libwebp --enable-ffplay --enable-vulkan --enable-libshaderc --pkg-config=/d/bld/ffmpeg_1773007679189/_build_env/Library/bin/pkg-config
libavutil 60. 8.100 / 60. 8.100
libavcodec 62. 11.100 / 62. 11.100
libavformat 62. 3.100 / 62. 3.100
libavdevice 62. 1.100 / 62. 1.100
libavfilter 11. 4.100 / 11. 4.100
libswscale 9. 1.100 / 9. 1.100
libswresample 6. 1.100 / 6. 1.100
[rawvideo @ 00000230F0BD3FC0] Estimating duration from bitrate, this may be inaccurate
Input #0, rawvideo, from 'C:\Users\runneradmin\AppData\Local\Temp\pytest-of-runneradmin\pytest-0\test_video_encoder_against_ffm14\temp_input.raw':
Duration: 00:00:01.00, start: 0.000000, bitrate: 663552 kb/s
Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 1280x720, 663552 kb/s, 30 tbr, 30 tbn
[out#0/flv @ 00000230F0B97FC0] Codec AVOption preset (Encoding preset) has not been used for any stream. The most likely reason is either wrong type (e.g. a video option with no video streams) or that it is a private option of some decoder which was not actually used for any stream.
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo (native) -> flv1 (flv))
Press [q] to stop, [?] for help
Output #0, flv, to 'C:\Users\runneradmin\AppData\Local\Temp\pytest-of-runneradmin\pytest-0\test_video_encoder_against_ffm14\ffmpeg_output.flv':
Metadata:
encoder : Lavf62.3.100
Stream #0:0: Video: flv1 ([2][0][0][0] / 0x0002), yuv420p(tv, progressive), 1280x720, q=2-31, 200 kb/s, 30 fps, 1k tbn
Metadata:
encoder : Lavc62.11.100 flv
Side data:
cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: N/A
[out#0/flv @ 00000230F0B97FC0] video:334KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 0.201926%
frame= 30 fps=0.0 q=31.0 Lsize= 335KiB time=00:00:01.00 bitrate=2743.2kbits/s speed=9.94x elapsed=0:00:00.10
=========================== short test summary info ===========================
FAILED test/test_encoders.py::TestVideoEncoder::test_video_encoder_against_ffmpeg_cli[30-to_file-encode_params2-flv] - json.decoder.JSONDecodeError: Expecting ',' delimiter: line 122 column 10 (char 2412)
=========== 1 failed, 1217 passed, 596 skipped in 525.29s (0:08:45) ===========
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in test/test_encoders.py with validate_frames_properties and _get_frames_info, then compare them with _get_video_metadata, which uses ffprobe's default format. Reproduce the failures on Windows with FFmpeg 8 and inspect the output consumed by json.loads. Done means the audio and video encoder CLI comparisons no longer produce flaky JSONDecodeError failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 62/100