undefined behavior in gcd when parameters are minimum values of signed integers
- Dominant language
- C++
- Stars
- 19
- Forks
- 58
- PR merge metrics
- No merged PRs in 30d
Description
This is being reported to me via a clang-analyzer check, and it refers to the gcd template function in common_factor_rt.hpp
Integer template parameter type is a signed integer, and both parameters ('a' and 'b') are std::numeric_limits\::min.
```
444 template
445 inline BOOST_CXX14_CONSTEXPR Integer gcd(Integer const &a, Integer const &b) BOOST_GCD_NOEXCEPT(Integer)
446 {
447 if(a == (std::numeric_limits::min)())
448 return a == static_cast(0) ? gcd_detail::gcd_traits::abs(b) : boost::integer::gcd(static_cast(a % b), b);
449 else if (b == (std::numeric_limits::min)())
450 return b == static_cast(0) ? gcd_detail::gcd_traits::abs(a) : boost::integer::gcd(a, static_cast(b % a));
451 return gcd_detail::optimal_gcd_select(static_cast(gcd_detail::gcd_traits::abs(a)), static_cast(gcd_detail::gcd_traits::abs(b)));
452 }
```
In the initial invocation, the condition on 447 is true, and the condition on 448 is false. This leads to a recursive call to the function with parameters a=0 and b=std::numeric_limits\::min.
In the recursive call, the condition on 447 is false, the condition on 449 is true, and the condition on 450 is false. Another recursive call is on-deck with a=0 and b=b%a, But a is zero at this point. Anything mod 0 is undefined behavior.
My suggested fix is to check if a and b are the same value at the start of the function, and return a right away.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.