jd / jd/tenacity

Statistics can be incorrectly initialized inside Temporal contexts

Open
#507 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
8.8k
Forks
359
Avg merge
1m
Merged PRs (30d)
1

Description

Within [temporal](https://docs.temporal.io/develop/python), an asynchronous python workflow manager, statistics are not being correctly initialized. Given that statistics are on a local thread, my best guess is that a different thread is calling "next_action" than the one that initializes the AsyncRetrying object.

Temporal has a custom AsyncEventLoop, and a function to suspend a job and then "replay" it up to a point, which further complicates its interactions here.

My proposed solution is to either make statistics a normal attribute (I assume this would be pretty delicate), or to ensure we handle uninitialized elements of "statistics" robustly (see PR below)

```
{
<...rest of big nested stack trace...>
raise ApplicationError(
"cause": {
"message": "'idle_for'",
"stackTrace": " File "/app/venv/lib/python3.10/site-packages/ml/features/workflows.py", line 509, in online_ingest_feature_source
async for attempt in retry_context:

File "/app/venv/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 166, in __anext__
do = await self.iter(retry_state=self._retry_state)

File "/app/venv/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 153, in iter
result = await action(retry_state)

File "/app/venv/lib/python3.10/site-packages/tenacity/_utils.py", line 99, in inner
return call(*args, **kwargs)

File "/app/venv/lib/python3.10/site-packages/tenacity/__init__.py", line 428, in next_action
self.statistics["idle_for"] += sleep
",
"applicationFailureInfo": {
"type": "KeyError"
}
},
"applicationFailureInfo": {}
},
"applicationFailureInfo": {}
},
"applicationFailureInfo": {}
}
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.