agronholm / agronholm/anyio

Logging to stdout from anyio worker processes

Open
#516 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
260
Avg merge
2d 8h
Merged PRs (30d)
19

Description

We wanted to use `anyio.to_process.run_sync` in order to execute CPU intensive code in worker processes, and we want to emit logs from the code executed by the worker processes. AnyIO uses the stdout/stderr of the worker processes for internal purposes to communicate with the parent process, which means we need another way to write to main process stdout from the worker process without race conditions, or an alternative way to write logs from inside Docker.

**Alternatives examined without changes to AnyIO**

1. os.dup the parent process stdout file descriptor and pass that file descriptor to the worker process. This will not work as expected since there can be race conditions if both the parent process and worker process write logs at the same time to the parent's stdout even via different file descriptors.
2. Write to Docker logs without using writing to stdout/stderr - we didn't find a way to do it, if you know any that will be appreciated.

**Change proposal to AnyIO**
Instead of hard-coding stdout/stdin as the communication mechanism between parent process and child process, allow AnyIO users to opt into another piping mechanism so stdout can still be used in the child process for e.g. logging purposes inside Docker.

Contributor guide

Open the contributing guide

Research direction

Start by reading anyio.to_process.run_sync and the worker-process stdout/stderr handling described in the issue. Define a safe alternative communication mechanism that preserves parent-child communication while allowing worker logging; done means the mechanism is configurable and avoids stdout race conditions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.