Non Blocking Subscription Tasks

Open
#299 0 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
22/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
python
Domain
backend, databases, networking

Research direction

Start by tracing the subscription task scheduler and the synchronous round-robin execution described in the issue. Compare the proposed green-thread, asyncio, and subprocess approaches, preserving SQLite connections in the parent process; done means independent subscription I/O no longer blocks other tasks while database safety is maintained.

Written by the indexing model from the issue text.

Description

feature-request system:downloading

Currently all subscriptions are run synchronously sequentially in a round robin way. This means that if any I/O is occuring in a subscription it ties up all other subscription tasks. This becomes especially egregious in either the replying server or the file storage medium has a slow response.

In general this has two consequences which are undesirable:

  • The database can sit unused while another subscription is performing network tasks
  • Requests which should have been already made will be delayed until the preceding task has finished (a server can calculate a response while hydrus is doing other things, especially if the requests are to different servers so bandwidth throttling is not a concern)

Therefore in general either

  • An entire subscription task should run on a green thread (clunky but easy thread multiplexing),or
  • The system should use asyncio (very granular thread multiplexing, but a pain to work with in preexisting code),0r
  • Network jobs should be sent to a subprocess pool (multiprocessing threading) NB DO NOT to move an open sqlite connection across processes, instead download move and calculate hash in a subprocess, then return results to the parent.

The first option is easy, the third option has the added benefit of true multithreading your hashing, but needs a little extra thought put into process management and IPC-queueing. I can't recommend the second option unless hydrus is already using an aio eventloop somewhere.

Dominant language
Python
Stars
3.2k
Forks
207
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from hydrusnetwork/hydrus

All issues in hydrusnetwork/hydrus

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.