librelane / librelane/librelane
Make logging multithread-friendly
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 543
- Forks
- 105
- Avg merge
- 5d 14h
- Merged PRs (30d)
- 4
Description
Description
LibreLane can technically run multiple flows in parallel using ThreadPoolExecutor - it's very I/O bound. The issue is with the logging module: All threads get to log to stdout by default.
The only way around this is to add this preamble to basically any script doing multiple flows:
options.set_show_progress_bar(False) # <- one progress bar per flow otherwise
set_log_level(LogLevels.WARNING) # self-explanatory, otherwise you get literally all terminal outputs across N steps
options.set_condensed_mode(True) # otherwise you get the terminal-wide rules per steps
However, this does not fix another problem, which is that flows add handles to the global logger for their .log files. So all errors from across all flows get added to error.log, for example. This is the error.log of one design that happened to be running alongside two other designs that failed LVS:
15 LVS errors found. - deferred
3 LVS errors found. - deferred
Proposal
It should be far easier to multithread with LibreLane. There should be a more well-thought-out relationship between the flow and the logger.
Maybe messages can be filtered by thread? Or tagged with the design name + run tag somehow?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how flows configure the global logging module, stdout output, progress bars, and per-flow .log handlers when used through ThreadPoolExecutor. Compare concurrent runs using distinct design names or run tags, and define done as keeping each flow's messages and error logs isolated while preserving useful terminal output without the listed preamble.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100