intel / intel/QAT_Engine

memory corruption when using qat service

Open
#79 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
443
Forks
137
PR merge metrics
No merged PRs in 30d

Description

Hello!

- qat driver version: 1.7.l.4.2.0-00012
- qat engine version: v0.5.37
- linux kernel version: 4.9.0

qat driver configure:

```
./configure --enable-icp-hb-fail-sim
```

qat engine configure:

```
./configure --with-openssl_dir=/path/to/openssl --with-openssl_install_dir=/path/to/openssl_install --with-qat_dir=/path/to/qat --enable-usdm --enable-upstream_driver
```

lsmod (removes the irrelevant entries):

```
qat_dh895xcc 6586 1
usdm_drv 82195 19
intel_qat 152264 56 qat_dh895xcc,usdm_drv
sha512_generic 4943 0
uio 7430 37 intel_qat
dh_generic 2535 1 intel_qat
authencesn 4740 0
authenc 4119 2 intel_qat,authencesn
rsa_generic 8940 1 intel_qat
mpi 12055 2 dh_generic,rsa_generic
asn1_decoder 2882 1 rsa_generic
```
In addition, we closed the boot option `intel_iommu`.

We are using the QAT dh895xcc series card for our own OpenResty/Nginx service. While we pouring traffic the Nginx with qat service, it crashes crazy. Some backtraces like:

```
2018/07/30 19:15:56 [alert] 273240#273240: backtrace: [00] nginx: worker process(ngx_backtrace+0x1f) [0x4595a4]
2018/07/30 19:15:56 [alert] 273240#273240: backtrace: [01] nginx: worker process() [0x459838]
2018/07/30 19:15:56 [alert] 273240#273240: backtrace: [02] /lib64/libpthread.so.0(+0xf7e0) [0x7f7d1b3d67e0]
2018/07/30 19:15:56 [alert] 273240#273240: backtrace: [03] [0x7f7d44fac29d]
src/tcmalloc.cc:283] Attempt to free invalid pointer 0x1
```

```
2018/07/30 19:15:50 [alert] 273235#273235: worker process 273235 exited on signal 11 (core dumped)
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [00] nginx: worker process(ngx_backtrace+0x1f) [0x4595a4]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [01] nginx: worker process() [0x459838]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [02] /lib64/libpthread.so.0(+0xf7e0) [0x7f7d1b3d67e0]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [03] /usr/local/marco/luajit/lib/libluajit-5.1.so.2(+0x2bedd) [0x7f7d1af42edd]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [04] /usr/local/marco/luajit/lib/libluajit-5.1.so.2(+0x30fdf) [0x7f7d1af47fdf]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [05] /usr/local/marco/luajit/lib/libluajit-5.1.so.2(+0x46851) [0x7f7d1af5d851]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [06] /usr/local/marco/luajit/lib/libluajit-5.1.so.2(+0xa736) [0x7f7d1af21736]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [07] /usr/local/marco/luajit/lib/libluajit-5.1.so.2(+0x4714f) [0x7f7d1af5e14f]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [08] /usr/local/marco/luajit/lib/libluajit-5.1.so.2(+0x14681) [0x7f7d1af2b681]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [09] /usr/local/marco/luajit/lib/libluajit-5.1.so.2(+0xbdaa) [0x7f7d1af22daa]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [10] /usr/local/marco/luajit/lib/libluajit-5.1.so.2(lua_pcall+0x2d) [0x7f7d1af3137d]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [11] nginx: worker process() [0x554056]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [12] nginx: worker process(ngx_http_lua_cache_loadbuffer+0x4e) [0x55422c]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [13] nginx: worker process(ngx_http_lua_filter_set_by_lua_inline+0x8b) [0x549bd4]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [14] nginx: worker process() [0x52c2d9]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [15] nginx: worker process() [0x4debc1]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [16] nginx: worker process(ngx_http_core_rewrite_phase+0x21) [0x470a40]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [17] nginx: worker process(ngx_http_core_run_phases+0x90) [0x47093b]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [18] nginx: worker process(ngx_http_handler+0x1b1) [0x4708a9]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [19] nginx: worker process(ngx_http_process_request+0x314) [0x47ef5b]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [20] nginx: worker process() [0x47d9f6]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [21] nginx: worker process() [0x47cf83]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [22] nginx: worker process() [0x47bd7b]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [23] nginx: worker process() [0x45eb61]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [24] nginx: worker process(ngx_process_events_and_timers+0xd3) [0x44f29b]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [25] nginx: worker process() [0x45c42b]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [26] nginx: worker process(ngx_spawn_process+0x656) [0x458eda]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [27] nginx: worker process() [0x45b570]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [28] nginx: worker process(ngx_master_process_cycle+0x296) [0x45ac59]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [29] nginx: worker process(main+0x564) [0x41cef0]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [30] /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f7d1a160d5d]
2018/07/30 19:15:50 [alert] 273235#273235: backtrace: [31] nginx: worker process() [0x41c799]
```

The first backtrace, shows that we are attempting to free the pointer where the address is invalid (0x1).
The second backtrace, after my own analysis, it crashes when LuaJIT is restoring the stack snapshot (back to interpreter), I have also sent email to luajit community for this issue, by the way, when I disable the JIT compiler, this type of segmentation fault disappears.

After I disable the qat service, our service works well. I don't know wether the qat service causes some memory corruption.

Is there any idea for this issue?

Regards
Alex Zhang

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.