iovisor / iovisor/bcc

Incorrect result while running biolatency.py with flags option on kernel 4.9.266

Open
#3,587 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
22.7k
Forks
4.1k
Avg merge
10d 4h
Merged PRs (30d)
3

Description

**Environment:** Linux kernel 4.9.266 on aarch64 platform
**Issue:** There is no "Write" operation while running biolatency.py with flags option on kernel 4.9.266, the workload is `dd if=/dev/zero of=/mnt/sda1 bs=1M count=64`, but it works fine with Linux kernel 5.10.46 on the same platform

The incorrect result attached at the end.

I have made some investigation and find the root cause.
Linux kernel 4.10 and later, the definition of `cmd_flags` of `struct request` in `include/linux/blkdev.h` is changed by [patchset](https://lore.kernel.org/linux-block/1476969135-32732-7-git-send-email-hch@lst.de/).
- 5.10.46
- https://elixir.bootlin.com/linux/v5.10.46/source/include/linux/blk_types.h#L332
- 4.9.266
- https://elixir.bootlin.com/linux/v4.9.266/source/include/linux/blk_types.h#L248
```
╔════════════════════════════════════╗
║4.9.x and earlier struct request -> ║
║ cmd_flags ║
╚════════════════════════════════════╝
┌────────┬─────────────────────────────────────────────────────────────────┐
│ req_op │ req_flag │
└────────┴─────────────────────────────────────────────────────────────────┘
▶──────◀ ▶───────────────────────────────────────────────────────────────◀
3 bits 61 bits


╔═════════════════════════════════════════════════════════════╗
║ 4.10.x and later ║
║ struct request -> cmd_flags ║
╚═════════════════════════════════════════════════════════════╝
┌────────────────────────────────────────────────────────┬─────────────────┐
│ req_flag │ req_op │
└────────────────────────────────────────────────────────┴─────────────────┘
▶──────────────────────────────────────────────────────◀ ▶───────────────◀
56 bits 8 bits

```
There is a fix (https://github.com/iovisor/bcc/pull/889) for bcc tool (biosnoop, biotop) related to this kernel changes.
The implementation of biolatency is passing the `cmd_flags` of `struct reqeust` back to user space, and Python part (https://github.com/iovisor/bcc/blob/master/tools/biolatency.py#L183) to analyze the meaning of cmd_flags.

I have made a local changes for biolatency to fit the data layout of cmd_flags for Linux kernel 4.9, and everything works correctly, but I am not sure how to make a coexist changes to fit both different kernel data layout.

For req_op, maybe we can reference to the changes of biosnoop/biotop to extract req_op in BPF program, but the definition of req_flags is totally different between two data layout.

I am wondering there is any more correct way to make a fix for this issue. Could you please give me some advice for it? Thank you.

4.9.266
```
(bcc)root@OpenWrt:/# biolatency -e -T -F
Tracing block device I/O... Hit Ctrl-C to end.
^C
15:58:50

flags = Idle-Metadata-Read
usecs : count distribution
0 -> 1 : 0 | |
2 -> 3 : 0 | |
4 -> 7 : 0 | |
8 -> 15 : 0 | |
16 -> 31 : 0 | |
32 -> 63 : 0 | |
64 -> 127 : 0 | |
128 -> 255 : 29 |************************************ |
256 -> 511 : 0 | |
512 -> 1023 : 0 | |
1024 -> 2047 : 32 |****************************************|
2048 -> 4095 : 12 |*************** |
4096 -> 8191 : 2 |** |
8192 -> 16383 : 0 | |
16384 -> 32767 : 0 | |
32768 -> 65535 : 0 | |
65536 -> 131071 : 1 |* |

flags = Read
usecs : count distribution
0 -> 1 : 0 | |
2 -> 3 : 0 | |
4 -> 7 : 0 | |
8 -> 15 : 0 | |
16 -> 31 : 0 | |
32 -> 63 : 0 | |
64 -> 127 : 7 |** |
128 -> 255 : 24 |********* |
256 -> 511 : 49 |******************* |
512 -> 1023 : 99 |****************************************|
1024 -> 2047 : 11 |**** |

flags = Idle-NoMerge-Metadata-Read
usecs : count distribution
0 -> 1 : 0 | |
2 -> 3 : 0 | |
4 -> 7 : 0 | |
8 -> 15 : 0 | |
16 -> 31 : 0 | |
32 -> 63 : 0 | |
64 -> 127 : 0 | |
128 -> 255 : 0 | |
256 -> 511 : 0 | |
512 -> 1023 : 0 | |
1024 -> 2047 : 0 | |
2048 -> 4095 : 0 | |
4096 -> 8191 : 0 | |
8192 -> 16383 : 0 | |
16384 -> 32767 : 0 | |
32768 -> 65535 : 2 |* |
65536 -> 131071 : 61 |****************************************|

flags = NoWait-Background-Idle-Priority-Read
usecs : count distribution
0 -> 1 : 0 | |
2 -> 3 : 0 | |
4 -> 7 : 0 | |
8 -> 15 : 0 | |
16 -> 31 : 0 | |
32 -> 63 : 0 | |
64 -> 127 : 4 |****************************************|
128 -> 255 : 2 |******************** |
256 -> 511 : 0 | |
512 -> 1023 : 0 | |
1024 -> 2047 : 0 | |
2048 -> 4095 : 1 |********** |

avg = 18477 usecs, total: 6208330 usecs, count: 336
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.