mgedmin / mgedmin/objgraph

I think _thread._local should be treated as a GC root

Open
#84 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement help-needed
Dominant language
Python
Stars
840
Forks
72
PR merge metrics
No merged PRs in 30d

Description

Writing this up because I'd been staring at "memory leaks" for a while that I think are actually downstream of TLS and finally (I believe) put the pieces together

From the docs:
> For simplicity’s sake let’s assume all of the roots are modules. objgraph provides a function, [is_proper_module()](https://objgraph.readthedocs.io/en/stable/objgraph.html#objgraph.is_proper_module), to check this. If you’ve any examples where that isn’t true, I’d love to hear about them (although see [Reference counting bugs](https://objgraph.readthedocs.io/en/stable/index.html#leaking-objects)).

I believe that I have another case of a GC "root" that is useful, `_thread._local`.

In the following code I dig through gc referrers for an object and it stops at `_thread._local`. `_thread._local` itself doesn't seem to have gc tracking so it doesn't link back to the module it's defined in unfortunately.

```
[0] % python test_threadlocal_theory.py
Found 1 instances of BB by by_type
DEPTH 0
{'big_boy': <__main__.BigBoy object at 0x100bc70e0>}
DEPTH 1
{<_thread._localdummy object at 0x100ab9010>: {'big_boy': <__main__.BigBoy object at 0x100bc70e0>}}
DEPTH 2
<_thread._local object at 0x100b334c0>
DEPTH 3
DEPTH 4
DEPTH 5
DEPTH 6
DEPTH 7
DEPTH 8
DEPTH 9
```

I think thread-locals will generally be dead-ends with the backref traversal algorithm, and having `objgraph.probable_gc_roots` might be useful?

```
#!/usr/bin/env python3
import gc
import threading
import objgraph

class BigBoy:
pass

tls = threading.local()

tls.big_boy = BigBoy()

bb_count = len(objgraph.by_type("BigBoy"))

print(f"Found {bb_count} instances of BB by by_type")

import threading as _threading

queue = [objgraph.by_type("BigBoy")[0]]
next_queue = []

ignore = [queue, next_queue, locals()]

for depth in range(10):
print(f"DEPTH {depth}")
while queue:
elt = queue.pop()
for ref in gc.get_referrers(elt):
if objgraph.is_proper_module(ref):
continue
if ref in ignore:
continue
print(ref)
next_queue.append(ref)
queue.extend(next_queue)
while next_queue:
next_queue.pop()

```

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no implementation file or test; start by running the supplied test_threadlocal_theory.py reproducer and tracing the back-reference traversal around is_proper_module(). Define the expected handling of _thread._local, then verify the chosen behavior with coverage for the reproducer.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devtools
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.