librelane / librelane/librelane

Make logging multithread-friendly

Open
#889 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

⛓️‍💥 breaking change ✨ enhancement
Dominant language
Python
Stars
543
Forks
105
Avg merge
5d 14h
Merged PRs (30d)
4

Description

Description

LibreLane can technically run multiple flows in parallel using ThreadPoolExecutor - it's very I/O bound. The issue is with the logging module: All threads get to log to stdout by default.

The only way around this is to add this preamble to basically any script doing multiple flows:

options.set_show_progress_bar(False) # <- one progress bar per flow otherwise
set_log_level(LogLevels.WARNING) # self-explanatory, otherwise you get literally all terminal outputs across N steps
options.set_condensed_mode(True) # otherwise you get the terminal-wide rules per steps

However, this does not fix another problem, which is that flows add handles to the global logger for their .log files. So all errors from across all flows get added to error.log, for example. This is the error.log of one design that happened to be running alongside two other designs that failed LVS:

15 LVS errors found. - deferred
3 LVS errors found. - deferred
Proposal

It should be far easier to multithread with LibreLane. There should be a more well-thought-out relationship between the flow and the logger.

Maybe messages can be filtered by thread? Or tagged with the design name + run tag somehow?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how flows configure the global logging module, stdout output, progress bars, and per-flow .log handlers when used through ThreadPoolExecutor. Compare concurrent runs using distinct design names or run tags, and define done as keeping each flow's messages and error logs isolated while preserving useful terminal output without the listed preamble.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.