Image decoders and encoders: improvements and changes vs torchvision

Start with the issue's Image decoders and Image encoders sections, using the listed TorchCodec-versus-torchvision API changes as the source material. Identify the repository's documentation entry point, which is not named here, and add a migration guide covering the documented breaking changes and improvements. Done means the guide is complete and linked from both TODO locations.

We migrated the image decoders and encoders from torchvision and torchvision-extra-decoders into TorchCodec. The TorchCodec implementation is more robust, supports more features, irons out rough edges, and is on par or better than Torchvision in terms of perf (decoding benchmarks, encoding benchmark).

TODO link migration guide.

Below is a list of changes and improvements. Generated by claude, checked by me.

Image decoders

BC-breaking API changes (vs torchvision)

• mode default is now RGB (was UNCHANGED)
• the apply_exif_orientation parameter is gone (EXIF is always applied)
• output dtype is now always uint8 by default (torchvision behavior can be replicated with output_dtype='auto')

Improvements: cross-cutting across all codecs

• All 5 modes supported for every codec (UNCHANGED/GRAY/GRAY_ALPHA/RGB/RGB_ALPHA). torchvision only does GRAY for PNG/JPEG; others reject/ignore it.
• Sequence/animation decoding to (N,C,H,W) for WebP, GIF, AVIF, HEIC. torchvision rejects animated WebP; extra-decoders errors on multi-image AVIF and decodes only the primary HEIC image.
• EXIF orientation applied by default (JPEG CPU+CUDA, PNG, WebP, AVIF; HEIC via libheif). torchvision is opt-in, PNG/JPEG only, and its CUDA path ignores EXIF.
• output_dtype control (uint8/uint16/"auto") on every decoder. torchvision has none as dtype is dictated by the source.
• decode_image auto-detects all six formats (incl. AVIF/HEIC); torchvision only four.
• Richer inputs: str/Path/bytes/bytearray/Tensor everywhere; non-contiguous input accepted (torchvision raises); list/batch input on CPU and CUDA.
• non-contiguous encoded input is accepted instead of raising.
• No separate package needed. AVIF works out of the box (libavif bundled). HEIC works if libheif is found at runtime (not bundled as it's LGPL). Install it yourself, else a clear ImportError). torchvision needs the standalone torchvision-extra-decoders for both, and decode_image couldn't dispatch to them.

Improvements: per-codec

• JPEG (CPU): truncated files error instead of returning garbage; correct CMYK/YCCK in UNCHANGED (PIL parity)
• JPEG (CUDA): there can now be more than one nvjpeg decoder per process, maximizing decoding throughput on multi-threaded pipelines.
• PNG: tRNS transparency (torchvision was TODO); palette images fixed (torchvision corrupts RGB_ALPHA/GRAY_ALPHA and returns raw indices for UNCHANGED); real libpng error messages.
• WebP: animated decoding; real grayscale (torchvision warns + returns color).
• GIF: Pillow-aligned rewrite — real RGBA/transparency, and disposal bug fixes (off-by-one disposal, DISPOSE_BACKGROUND over-clear, missing DISPOSE_PREVIOUS, uninitialized memory for oversized first frame).
• AVIF: progressive stills no longer error; num_threads for parallel decode.
• HEIC: correct full-range >8-bit→uint16 (extra-decoders never reaches full white); big-endian correct; clear missing-libheif / missing-codec errors.
• Performance: nvJPEG per-device decoder pool + lock-free decode (vs single global mutex), native per-image channels, mixed Y/RGB batches; ChannelsLast outputs; detection + file reads moved to Python (negligible cost, fewer ops).
• Safety: JPEG setjmp/longjmp UB segfault fixed (wasn't happening in TV due to O2 opt mode, but still a bug fix); PNG no longer throws C++ exceptions through libpng's C stack; contiguity/CPU-checked validation.

Image encoders

TODO link migration guide

BC-breaking API changes

• Functional API replaced by encoder classes: JpegEncoder(img).to_file(...) instead of encode_jpeg/write_jpeg. Same for PNG.
• List/batch input for the jpeg CUDA encoder is gone. encode_jpeg accepted a list of tensors (and used the batch path on CUDA). Benchmarks show that this isn't needed with the new implementation.

Improvements:

• Native to_file(): bytes are streamed straight to the file through the C++ IOInterface, instead of torchvision's encode-to-tensor-then-write_file two-step (no full encoded buffer materialized as a tensor first). This is faster.
• Native to_file_like(): any writable object with write/seek (io.BytesIO, an open binary file, …). torchvision has no file-like support anywhere.
• CUDA destinations all work: to_file/to_file_like copy the encoded bytes back to CPU; to_tensor keeps them on the GPU (deliberately, e.g. for cuFile), matching torchvision's GPU-tensor return.
• JPEG CUDA: there can now be more than one nvjpeg encoder per process, maximizing encoding throughput on multi-threaded pipelines