pyronear / pyronear/pyro-engine

Anonymizer stream survives decoder death and can never be stopped until reboot

Open
#411 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
19
Forks
8
PR merge metrics
No merged PRs in 30d

Description

Body:

While analyzing a year of mediamtx logs, I found 3 streams from betschdorf (one of the few sites with "anonymizer": true) that ran for 14h, 4h and 6.9h with no viewer, all ending exactly at 22:00 UTC, the nightly reboot time. Everything else in the fleet stops within minutes.

Root cause, two combined issues in pyro_camera_api:

  1. The encoder outlives the decoder. In anonymizer mode the stream is a decoder + encoder thread pair. The encoder loop (services/anonymizer_rtsp.py) reads self._frames.get(), which returns the last stored frame, and pushes it at fixed FPS. If the decoder dies (RTSP hiccup from the camera), the encoder keeps streaming the frozen frame to mediamtx indefinitely.

  2. Every stop path skips a half-dead pipeline. stop_any_running_stream (services/stream.py) only stops a pipeline when is_pipeline_running() is true, which requires both threads alive:

if is_pipeline_running(pipeline): # decoder AND encoder alive

With the decoder dead this returns false, so the pipeline is skipped by the /stop_stream endpoint (which then answers "No active stream was running" to the operator), by the 120s idle stopper, and by the cleanup before starting a new stream. Nothing can stop the zombie encoder except the nightly reboot.

Suggested fix:

  • In stop_any_running_stream, stop the pipeline if either thread is alive (both stop() calls are idempotent):
    if is_thread_alive(pipeline.decoder) or is_thread_alive(pipeline.encoder):
  • Defense in depth: have the decoder signal the encoder to stop when it exits, so a camera glitch freezes the stream for seconds instead of hours.

Verified present in all deployed versions (v1.0.11 through v1.0.18); the idle stopper itself works fine, it is just bypassed by the AND condition.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with stop_any_running_stream in services/stream.py and the encoder loop in services/anonymizer_rtsp.py, then trace the /stop_stream endpoint, idle stopper, and cleanup path. Verify that a pipeline with only one live thread is stopped, and that decoder exit signals the encoder to stop so frozen streams do not persist.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.