iovisor / iovisor/bcc

memleak: could not determine address of symbol malloc

Open
#3,432 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
C
Stars
22.7k
Forks
4.1k
Avg merge
5d 13h
Merged PRs (30d)
3

Description

I recently discovered a memory leak issue in `php7.3-fpm` on our production servers due to an outdated [Ioncube loader](https://www.ioncube.com/loaders.php). I wanted to see if I could discover the known issue using `memleak`, but I can't get it to work:

```
server ~ # memleak-bpfcc -p 9268
Attaching to pid 9268, Ctrl+C to quit.
Traceback (most recent call last):
File "/usr/sbin/memleak-bpfcc", line 416, in
attach_probes("malloc")
File "/usr/sbin/memleak-bpfcc", line 406, in attach_probes
pid=pid)
File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 982, in attach_uprobe
(path, addr) = BPF._check_path_symbol(name, sym, addr, pid)
File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 727, in _check_path_symbol
raise Exception("could not determine address of symbol %s" % symname)
Exception: could not determine address of symbol malloc
```

I thought that installing the `libc6-dbg` package should enable it to locate `malloc`, but apparently it doesn't.

Normal tracing works fine though:

```
server ~ # trace-bpfcc -p 9268 'c:malloc' | head
PID TID COMM FUNC
9268 9268 php-fpm7.3 malloc
9268 9268 php-fpm7.3 malloc
9268 9268 php-fpm7.3 malloc
9268 9268 php-fpm7.3 malloc
9268 9268 php-fpm7.3 malloc
9268 9268 php-fpm7.3 malloc
9268 9268 php-fpm7.3 malloc
9268 9268 php-fpm7.3 malloc
9268 9268 php-fpm7.3 malloc
```

```
server ~ # lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 10 (buster)
Release: 10
Codename: buster

server ~ # apt-cache policy php7.3-fpm
php7.3-fpm:
Installiert: 7.3.27-1~deb10u1
Installationskandidat: 7.3.27-1~deb10u1
Versionstabelle:
*** 7.3.27-1~deb10u1 500
500 http://deb.debian.org/debian buster/main amd64 Packages
500 http://deb.debian.org/debian-security buster/updates/main amd64 Packages
100 /var/lib/dpkg/status

server ~ # apt-cache policy libc6-dbg
libc6-dbg:
Installiert: 2.28-10
Installationskandidat: 2.28-10
Versionstabelle:
*** 2.28-10 500
500 http://deb.debian.org/debian buster/main amd64 Packages
100 /var/lib/dpkg/status
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure with memleak-bpfcc -p 9268 on the reported Debian 10 setup, then compare it with trace-bpfcc -p 9268 'c:malloc'. Start at the memleak-bpfcc attach_probes("malloc") entry point and the symbol lookup path shown in the traceback. Done means memleak can attach to malloc in this case and report the leak data without the address-resolution exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, python
Domain
observability-sre, operating-systems, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.