AcademySoftwareFoundation / AcademySoftwareFoundation/OpenColorIO

Regression: Excessive precision loss in NVIDIA Cg shaders

Open
#1,884 1 comment 0 reactions 0 assignees View on GitHub
Needs Discussion
Dominant language
C++
Stars
2.1k
Forks
503
PR merge metrics
No merged PRs in 30d

Description

Certain operations in the shader generation do not account for the use of `half` in the NVIDIA Cg shading language.

Excerpt of a NVIDIA Cg shader generated with OpenColorIO 1.1.1:
```
out_pixel.rgb = max(half3(6.10352e-05, 6.10352e-05, 6.10352e-05), half3(1, 1, 1) * out_pixel.rgb + half3(0, 0, 0));
out_pixel.rgb = half3(1.4427, 1.4427, 1.4427) * log(out_pixel.rgb) + half3(0, 0, 0);
```

Excerpt of a NVIDIA Cg shader generated with OpenColorIO 2.1.3 [using the same OCIO config]:
```
outColor.rgb = max( half3(0., 0., 0.), outColor.rgb);
outColor.rgb = log2(outColor.rgb);
```

Note that `6.10352e-05` (`GetHalfNormMin()`) is used in OCIO 1, whereas `0.` is now seen in OCIO 2.

The issue was initially noticed with images that use pure colors (e.g. #FF0000 or #00FF00): the color turns black.

| Input Image | OpenColorIO 1 | OpenColorIO 2 |
| --- | --- | --- |
| ![Input Image](https://github.com/AcademySoftwareFoundation/OpenColorIO/assets/33085886/fbc9bacc-32bb-4420-a0e3-bbb88a21ad17) | ![OpenColorIO 1](https://github.com/AcademySoftwareFoundation/OpenColorIO/assets/33085886/701bc38d-f12c-444c-881f-5de81ad4e79a) | ![OpenColorIO 2](https://github.com/AcademySoftwareFoundation/OpenColorIO/assets/33085886/6c4c3e48-d4cf-48c5-b765-3683a55f6d3d) |

The issue appears to originate in `src/OpenColorIO/ops/log/LogOpGPU.cpp`, with the unconditional use of `std::numeric_limits::min()`. A potential solution, that addresses the case exposed above, is:

```diff
- const float minValue = std::numeric_limits::min();
-
- GpuShaderText st(shaderCreator->getLanguage());
+ const GpuLanguage lang = shaderCreator->getLanguage();
+
+ const float minValue = lang == GPU_LANGUAGE_CG
+ ? GetHalfNormMin()
+ : std::numeric_limits::min();
+
+ GpuShaderText st(lang);
```

This is likely not the only oversight, though; there may be more precision issues of this nature that involve the Cg generator.

Contributor guide

Open the contributing guide

Research direction

Start in src/OpenColorIO/ops/log/LogOpGPU.cpp and inspect how shader generation handles minimum values for the Cg language, comparing the OCIO 1 and 2 shader excerpts in the issue. Check related Cg precision paths for similar assumptions; done means pure colors no longer turn black and the broader precision regression is covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
computer-graphics
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.