OpenMathLib / OpenMathLib/OpenBLAS

Incorrect result with `cblas_dgemv` vs reference netlib and other libraries

Open
#4,324 14 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
7.6k
Forks
1.7k
Avg merge
1d 3h
Merged PRs (30d)
42

Description

We recently switched to testing openBLAS on a project and are noticing some test case failures due to a matrix multiplication operation returning an incorrect result.

This issue has been observed on a variety of platforms (Ubuntu 22.04, RHEL7, RHEL9, MSYS2 mingw), a variety of compilers (clang-15, mingw-13, gcc-12, gcc-11, gcc-9), as well as a variety of openblas versions(0.3.3, 0.3.20, 0.3.21, 0.3.24), and a variety of CPUs:

  • Intel(R) Xeon(R) CPU E5-4650 0 @ 2.70GHz
  • Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz
  • Intel(R) Core(TM) i7-9850H CPU @ 2.60GHz 2.59 GHz
Reproduction

I have attached a minimally reproducible example (in C++) showing the problem

Reproduction Code 
#include <cblas.h>

#include <cassert>
#include <iostream>
#include <fstream>

constexpr static size_t size = 16;

void get_values(double* A, double* b, const char* binfile) {
    std::fstream binaryReader;
    binaryReader.open(binfile, std::ios::in | std::ios::binary);
    assert(binaryReader.is_open());

    for (size_t i = 0; i < (size * size); i++) {
        binaryReader.read(reinterpret_cast<char *>(&A[i]), sizeof(double));
    }

    for (size_t i = 0; i < size; i++) {
        binaryReader.read(reinterpret_cast<char *>(&b[i]), sizeof(double));
    }
}

void matrixMultiply(const double* A, const double* b, double* c) {
    const size_t N = size;
    for (size_t i = 0; i < N; ++i) {
        for (size_t j = 0; j < N; ++j){
            c[i] += A[ (i * N) + j] * b[j];
        }
    }
}

void CBlasMatrixMultiply(const double* a, const double* b, double* c){
    int32_t M = size;
    int32_t N = size;
    cblas_dgemv(CblasRowMajor,
                CblasNoTrans,
                M, N,
                1.0, a, N,
                b, 1,
                1.0, c, 1);
}

int main(int argc, char* argv[]) {
    auto* A = new double[256]();
    auto* b = new double[16]();

    const char* binfile = argc > 1 ? argv[1] : BIN_FILE;
    get_values(A, b, binfile);

    auto* c_blas = new double[16]();
    CBlasMatrixMultiply(A, b, c_blas);

    auto *c_mat = new double[16]();
    matrixMultiply(A, b, c_mat);

    printf("           BLAS                         MAT\n");
    for (size_t i = 0; i < size; i++) {
        printf("%0.18e\t%0.18e\n", c_blas[i], c_mat[i]);
    }

    // dot product?
    auto c_ddot = cblas_ddot(size, A + (size * 15), 1, b, 1);
    printf("ddot: %0.18e\n", c_ddot);
    printf("dgemv: %0.18e\n", c_blas[15]);
    printf("ddot == dgemv? %s\n", c_ddot == c_blas[15] ? "YES" : "NO");
    printf("dgemv is 0.0? %s\n", c_blas[15] == 0.0 ? "YES" : "NO");

    delete[] A;
    delete[] b;
    delete[] c_blas;
    delete[] c_mat;

    return 0;
}

Compile this code with:

g++ -o blas_test blas_test.cpp -DBIN_FILE=\"path/to/bin\" $(pkg-config --libs --cflags openblas)

And observe the following output:

OpenBLAS result 
           BLAS                         MAT
1.175201193643801378e+00        1.175201193643801822e+00
1.103638323514327002e+00        1.103638323514327224e+00
3.578143506473725477e-01        3.578143506473724922e-01
7.045563366848892062e-02        7.045563366848883735e-02
9.965128148869371871e-03        9.965128148869309421e-03
1.099586127207577424e-03        1.099586127207556390e-03
9.945433911373591229e-05        9.945433911360671605e-05
7.620541308983597162e-06        7.620541308896932986e-06
5.064719745540013918e-07        5.064719744437437483e-07
2.971814122565419325e-08        2.971814140421127963e-08
1.560886642160141946e-09        1.560886564391797127e-09
7.419920233786569952e-11        7.419902813631537848e-11
3.221201083647429186e-12        3.221406076314455686e-12
1.292299600663682213e-13        1.289813102624490508e-13
4.440892098500626162e-15        4.440892098500626162e-15
0.000000000000000000e+00        -4.440892098500626162e-16
ddot: 0.000000000000000000e+00
dgemv: 0.000000000000000000e+00
ddot == dgemv? YES
dgemv is 0.0? YES

It is worth noting that only the last value is different outside of acceptable numerical precision, and that every other value passes within 1e-16. Furthermore, a value of exactly 0.0 is, in itself, suspicious, as there's no real circumstance the value could be that.

Change the compile command to:

# cblas here is netlib
g++ -o blas_test blas_test.cpp -DBIN_FILE=\"path/to/bin\" $(pkg-config --libs --cflags cblas)

and observe this result:

netlib result 
           BLAS                         MAT
1.175201193643801822e+00        1.175201193643801822e+00
1.103638323514327224e+00        1.103638323514327224e+00
3.578143506473724922e-01        3.578143506473724922e-01
7.045563366848883735e-02        7.045563366848883735e-02
9.965128148869309421e-03        9.965128148869309421e-03
1.099586127207556390e-03        1.099586127207556390e-03
9.945433911360671605e-05        9.945433911360671605e-05
7.620541308896932986e-06        7.620541308896932986e-06
5.064719744437437483e-07        5.064719744437437483e-07
2.971814140421127963e-08        2.971814140421127963e-08
1.560886564391797127e-09        1.560886564391797127e-09
7.419902813631537848e-11        7.419902813631537848e-11
3.221406076314455686e-12        3.221406076314455686e-12
1.289813102624490508e-13        1.289813102624490508e-13
4.440892098500626162e-15        4.440892098500626162e-15
-4.440892098500626162e-16       -4.440892098500626162e-16
ddot: -4.440892098500626162e-16
dgemv: -4.440892098500626162e-16
ddot == dgemv? YES
dgemv is 0.0? NO

Here is the binary file that contains a 16x16 matrix and a 16x1 vector:
(NOTE: This is a binary data file, extension changed to make github happy)
reproduction.txt

Other Notes

We have done extensive testing in other BLAS-like environments to get a result close to the expected -4e-16 result, which passes our test. Both MATLAB (2023a) and numpy (1.26 w/ MKL) return a result very close to what we expect, and pass our test. And, obviously, our naive matrix multiplication in the reproduction code gives

The matrix in question is not overly ill-conditioned, it has a condition number of ~10.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by compiling the attached C++ reproduction with OpenBLAS and netlib CBLAS using the commands in the issue, then inspect the cblas_dgemv and cblas_ddot results against the naive multiplication. Use reproduction.txt as the input and compare behavior across the reported environments. Done means identifying the cause of the final-element discrepancy and demonstrating corrected results without losing the stated numerical accuracy.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.