equinor / equinor/oneseismic-api
Floating point arithmetic and precision
- Dominant language
- C++
- Stars
- 5
- Forks
- 8
- PR merge metrics
- No merged PRs in 30d
Description
Our API, as well as OpenVDS's API operate primarily on floats. However, we do a lot of intermediate floating point arithmetic before we pass values along to OpenVDS. E.g. coordinate transformations. To avoid precision loss (accumulating precision loss) we do most of these calculations on double and convert back to float only when passing the values along to OpenVDS. However, [this](https://github.com/equinor/vds-slice/blob/1a921badf51bb6dba35953b82ce0d3189e5aa351/internal/core/cppapi_data.cpp#L362C9-L362C9) annotation to voxel transform computes on floats, and when used [here](https://github.com/equinor/vds-slice/blob/1a921badf51bb6dba35953b82ce0d3189e5aa351/internal/core/axis.hpp#L29) it introduces a precision loss to the surface depths we are fetching from OpenVDS.
To fix this we should do this annotation to voxel transformation on doubles
Contributor guide
Assessment
This issue has not been assessed yet.