Lightning-AI / Lightning-AI/LitServe
Handle case when Logger.process is stuck
@aniketmaurya is already working on this.
Since Sep 23, 2024.
- Dominant language
- Python
- Stars
- 3.9k
- Forks
- 304
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 6
Description
If `process` gets stuck here (e.g. a POST request hangs) for one of the loggers, logging hangs for all loggers. This will also cause `queue` to grow unbounded.
Unfortunately fixing this requires running `process` in separate processes.
A possible solution for the unbounded problem is having a loop that gets elements from the queue into a local `deque` (this could happen in a thread), from which `process` grabs items. This would help also fanning out to multiple loggers.
I don't want to necessarily complicate things right now, just calling this out because it's likely to happen.
_Originally posted by @lantiga in https://github.com/Lightning-AI/LitServe/pull/284#discussion_r1769581574_
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.