jmespath / jmespath/jmespath.py

`TreeInterpreter` creates reference cycle, causing GC pressure

Open
#291 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
2.4k
Forks
212
PR merge metrics
No merged PRs in 30d

Description

We recently noticed that a heavy JMESpath workload was triggering a large number of garbage collection runs. We are using `jmespath.compile()`, and we tracked this down to the `jmespath.visitor.TreeInterpreter` that is created on every call to `ParsedResult.search(): https://github.com/jmespath/jmespath.py/blob/bbe7300c60056f52413603cf3e2bcd0b6afeda3d/jmespath/parser.py#L508

It appears that `TreeInterpreter` creates a reference cycle, which leads to the GC being triggered frequently to clean up the cycles. As far as I can tell, the problem comes from the `Visitor._method_cache`: https://github.com/jmespath/jmespath.py/blob/bbe7300c60056f52413603cf3e2bcd0b6afeda3d/jmespath/visitor.py#L91-L93

...which store references to methods that are bound to `self` in a member of `self`.

## Possible solution

We worked around the problem by monkey patching `ParsedResult` so that it (1) caches a `default_interpreter` for use when `options=None`, and (2) uses it in `search()`. If I understand correctly, we could go further and use a global `TreeInterpreter` for all `ParsedResult` instances. The `TreeInterpreter` seems to be stateless apart from `self._method_cache` and that implementation seems to be thread-safe (with only the risk of multiple lookups for the same method in a multithreaded case).

I'd be happy to contribute a PR for either version if this would be welcome.

## How to reproduce

The following reproducer shows the problem:

```python
import jmespath

import gc
gc.set_debug(gc.DEBUG_COLLECTABLE)

pattern = jmespath.compile("foo")
value = {"foo": "bar"}

for _ in range(1000000):
pattern.search(value)
```

...where the output contains one million repetitions of something like:

```
gc: collectable
gc: collectable
gc: collectable
gc: collectable
gc: collectable
gc: collectable
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.