AcademySoftwareFoundation / AcademySoftwareFoundation/OpenImageIO

[BUG] Slow read performance for multichannel EXRs in oiiotool

Open
#4,707 5 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
C++
Stars
2.4k
Forks
698
Avg merge
2d 23h
Merged PRs (30d)
46

Description

**Describe the bug**

I'm getting very slow read performance on multi-channel EXRs where I just need to parse the RGBA channel and write out a `.png` file from it.

From 19s on my own machine (which is crazy for a 55 MB exr?) to 5s on the fastest machine I have around.
On my own slow machine, taking the same EXR (on no cold/hot cache) through Fusion to a PNG takes 1.3s

The 19 seconds (but 1.3s through fusion) is on:
- Windows 10
- 2x Intel(R) Xeon(R) CPU E5-2680 0 @2.70Ghz
- Dual Xeon in HP workstation
- 128 GB RAM
- 4 GB GPU

The 5 seconds is on:
- Windows 11
- AMD Ryzen 9 9900X 12-Core Processor (4.40 Ghz)
- 128 GB RAM
- 12 GB GPU

**OpenImageIO version and dependencies**

See test runs [here](https://gist.github.com/BigRoy/327f3c978b32182348c54ed9f30a5cbe).

```
OIIO 3.1.1.0dev | unknown arch?
Build compiler: MSVS 1943 | C++17/199711
HW features enabled at build: sse2
No CUDA support (disabled / unavailable at build time)
Dependencies: DCMTK NONE, expat 2.6.3, FFmpeg NONE, fmt 10.2.1, Freetype 2.13.2, GIF NONE, Imath 3.2.0, JPEG 80,
JXL NONE, libdeflate 1.23, Libheif NONE, libjpeg-turbo 3.0.4, LibRaw NONE, libuhdr 1.2.0, minizip-ng 4.0.7,
OpenColorIO 2.4.1, OpenCV NONE, OpenEXR 3.4.0, OpenJPEG NONE, OpenVDB NONE, PNG 1.6.47, Ptex NONE, Ptex NONE,
pystring 1.1.4, Robinmap 1.4.0, TBB 2022.2.0, TIFF 4.6.0, WebP 1.4.0, yaml-cpp 0.8.0, ZLIB 1.3.1, ZLIB 1.3.1
```

**To Reproduce**

Steps to reproduce the behavior:
1. `oiiotool -v -i:ch=R,G,B,A input.exr -o output.png`

I'm explicitly filtering `-i:ch=R,G,B,A` to avoid reading the rest of the file. Also adding `:now=1` doesn't make much of a difference.
Other options, like also passing `-ch` flag are included in the linked test runs with their timings. Using `i:ch=...` gave the fastest results, but still at 19s seems awful.

Here's the file I'm testing with: https://we.tl/t-Fjm4XVFw8z
It'll only remain available for about four weeks.

**Evidence**

See test runs [here](https://gist.github.com/BigRoy/327f3c978b32182348c54ed9f30a5cbe).

### Additional Context

This also came up on ASWF slack with a lengthy thread [here](https://academysoftwarefdn.slack.com/archives/C05782U3806/p1744995471959379)

Some notable comments from there:

#### Build options didn't make much of a difference?

> So, I've been testing some build variants.
> When enabling more hardware features like SSSE3, SSE4.1 SSE4.2, AVX etc. makes it so that the executable doesn't work on my machine - likely due to some AVX (it does work on newer machines but there was no real measurable performance difference on this simple read/write test cases) so I opted going back to just the sse2 default but adding OpenTBB which does run. I've added the performance on my slow machine to the gist (see link below)
> There is no noticable speed difference. It remains around 18/19 seconds on my machine

> For what it's worth, using `--frames 1001-1006` and `--parallel-frames` also takes about 20 seconds on my machine (so almost no time increase).
> Using `--frames 1001-1030` with `--parallel-frames` takes about 53 seconds on my machine.
>
> So I can get good gains with allowing it to read many files. However, it's still the majority the read performance taking a long time it seems.

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with the supplied EXR and `oiiotool -v -i:ch=R,G,B,A input.exr -o output.png`, then compare timings with the linked test runs. Investigate the multichannel EXR read path and the effect of channel filtering; done means the RGBA-only conversion no longer exhibits the reported excessive read time and the relevant timings are documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.