AcademySoftwareFoundation / AcademySoftwareFoundation/OpenColorIO
Regression: Excessive precision loss in NVIDIA Cg shaders
- Dominant language
- C++
- Stars
- 2.1k
- Forks
- 503
- PR merge metrics
- No merged PRs in 30d
Description
Certain operations in the shader generation do not account for the use of `half` in the NVIDIA Cg shading language.
Excerpt of a NVIDIA Cg shader generated with OpenColorIO 1.1.1:
```
out_pixel.rgb = max(half3(6.10352e-05, 6.10352e-05, 6.10352e-05), half3(1, 1, 1) * out_pixel.rgb + half3(0, 0, 0));
out_pixel.rgb = half3(1.4427, 1.4427, 1.4427) * log(out_pixel.rgb) + half3(0, 0, 0);
```
Excerpt of a NVIDIA Cg shader generated with OpenColorIO 2.1.3 [using the same OCIO config]:
```
outColor.rgb = max( half3(0., 0., 0.), outColor.rgb);
outColor.rgb = log2(outColor.rgb);
```
Note that `6.10352e-05` (`GetHalfNormMin()`) is used in OCIO 1, whereas `0.` is now seen in OCIO 2.
The issue was initially noticed with images that use pure colors (e.g. #FF0000 or #00FF00): the color turns black.
| Input Image | OpenColorIO 1 | OpenColorIO 2 |
| --- | --- | --- |
|  |  |  |
The issue appears to originate in `src/OpenColorIO/ops/log/LogOpGPU.cpp`, with the unconditional use of `std::numeric_limits::min()`. A potential solution, that addresses the case exposed above, is:
```diff
- const float minValue = std::numeric_limits::min();
-
- GpuShaderText st(shaderCreator->getLanguage());
+ const GpuLanguage lang = shaderCreator->getLanguage();
+
+ const float minValue = lang == GPU_LANGUAGE_CG
+ ? GetHalfNormMin()
+ : std::numeric_limits::min();
+
+ GpuShaderText st(lang);
```
This is likely not the only oversight, though; there may be more precision issues of this nature that involve the Cg generator.
Contributor guide
Research direction
Start in src/OpenColorIO/ops/log/LogOpGPU.cpp and inspect how shader generation handles minimum values for the Cg language, comparing the OCIO 1 and 2 shader excerpts in the issue. Check related Cg precision paths for similar assumptions; done means pure colors no longer turn black and the broader precision regression is covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- computer-graphics
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100