ReactiveX / ReactiveX/RxPY

Connectable observable sends events multiple times

Open
#591 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Python
Stars
5k
Forks
365
Avg merge
3d 14h
Merged PRs (30d)
4

Description

I'm using rxpy for a realtime audio processing tool. The tool receives two event streams, one that contains audio chunks and one that contains small text like annotation snippets. The flow looks similar to this:

audio ----> .publish() ----> rec ----> p1 ---> p2 ---> p 3 ---> p4 ---> merge ---> output
                                        \              /               /
                                          ---> p5 ----                /
annotations --------------------------------------------------> p6 ---

Here, the p<x> nodes are processing nodes (implemented in numpy/pytorch) and the rec node is a recorder, that writes its input to disk and otherwise passes it on unchanged. I'm using the .publish() call to support the branching that comes after the rec node.

When I open the audio file written by the rec node, every chunk has been written 3 times, which implies that the rec node received every chunk 3 times. Is this intended behaviour? How can I avoid this? I'm worried that downstream nodes (p1-p5) might receive multiple repetitions of the same chunk as well and therefore might not operate as intended. However, the pipeline as a whole seems to work correctly.

I tried a number of variations:

  1. No .publish() call: Events in the first pipeline get stuck right before p3. p4 never receives any events.
  2. Introduce a separate step before the rec node that drops events if they have the same md5 sum as the previous event (either using a combination of scan and filter or a filter with a class). This makes the audio file look ok, but the overall pipeline becomes prohibitively slow and is essentially broken.
  3. Move the .publish() call to a later stage: The results are essentially the same as 1.

Thank you for your help.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the .publish() call and recorder node described in the report, then reproduce the three-branch flow with the audio and annotation streams. Trace subscriptions and verify whether each chunk is written once, documenting the intended behavior and any relevant regression test entry point found in the repository.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python, pytorch
Domain
audio-video-rtc, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.