iovisor / iovisor/bcc

tcptop: incorrect bytes sent (tcp_sendmsg)

Open
#2,440 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
22.7k
Forks
4.1k
Avg merge
5d 13h
Merged PRs (30d)
3

Description

Hi I noticed that for the `tcptop.py` tool, binding to the `tcp_sendmsg` probe to count the bytes could sometimes results into miscounting when `tcp_sendmsg` fails (it can occur when it returns the `EAGAIN` error code for instance, it seems that this can happen [here](https://elixir.bootlin.com/linux/v4.8/source/net/core/stream.c#L172)).

In this case the `tcp_sendmsg` probe is triggered and the `size` parameter passed is used to increment the count of bytes sent without checking for an error.

Some solutions would be to:
- use a `kretprobe` on `tcp_sendmsg` to check for the returned value (# of bytes sent OR an error) but my understanding is that this can lead to missing `tcp_sendmsg` since there is a limit for multiple `kretprobe` running at the same time set by the `max_active` parameter.

- use [`tcp_transmit_skb`](https://elixir.bootlin.com/linux/v4.8/source/net/ipv4/tcp_output.c#L901) which has the # of bytes to be sent inside the sk_buff parameter. This would also help capture bytes sent via TCP retransmits, However it seems that tcp_transmit_skb do not have a reliable access to the PID (so this might require to remove the pid from the tuple ?)

Do you think there is a better way to fix this issue ?

### Reproducing the issue

The following was done on a Vagrant VM (`hashicorp-vagrant/ubuntu-16.04`):
```
$ uname -a
Linux vagrant 4.4.0-109-generic #132-Ubuntu SMP Tue Jan 9 19:52:39 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
```

I was able to reproduce it by setting the `so_sndtimeo` parameter to a really low value and trying to send a "big" payload.

Running the following command (which will open a local tcp connection and try to send 100Mb of `1`s to it should trigger this issue):

`nc -l 3000 & python3 script.py 100000`

Where `script.py` is:

```python
#!/usr/bin/env python
# -*- coding: utf-8 -*-

import socket
import struct
import sys

addr = "127.0.0.1"
port = 3000

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((addr, port))

# 1us per attempt
sec = 0
usec = 1

timeval = struct.pack('ll', sec, usec)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_SNDTIMEO, timeval)

data = (int(sys.argv[1]) * 1024 * "1").encode()

print("Going to send {} bytes".format(len(data)))

sent = sock.sendmsg([data])

print("Sent {} bytes ({} lost)".format(sent, len(data) - sent))
```

Output of `nc -l 3000 & python3 script.py 100000`

```
Going to send 102400000 bytes
Sent 6855898 bytes (95544102 lost)
```

Output of `sudo python tcptop.py -C`:

```
PID COMM LADDR RADDR RX_KB TX_KB
4809 4809 127.0.0.1:37004 127.0.0.1:3000 0 100000
4808 4808 127.0.0.1:3000 127.0.0.1:37004 6695 0
1816 sshd 10.0.2.15:22 10.0.2.2:50051 0 0
```

Expected:

```
PID COMM LADDR RADDR RX_KB TX_KB
4809 4809 127.0.0.1:37004 127.0.0.1:3000 0 6695
4808 4808 127.0.0.1:3000 127.0.0.1:37004 6695 0
1816 sshd 10.0.2.15:22 10.0.2.2:50051 0 0
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with tcptop.py and inspect how the tcp_sendmsg probe uses its size parameter when the call returns an error. Reproduce the mismatch with the provided Vagrant/Linux setup, nc command, and script.py, then compare the reported TX_KB with the sender's actual bytes. Done means failed sends are not counted and the reproducer reports approximately 6695 KB sent.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, python
Domain
networking, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.