MetPX / MetPX/sarrac

post_retry support for mirrorring

Open
#171 2 comments 0 reactions 0 assignees View on GitHub
enhancement mirroring question worries
Dominant language
C
Stars
4
Forks
1
Avg merge
5d 1h
Merged PRs (30d)
1

Description

The client has concerns about robustness of the mirroring post generation during broker outages. Currently, I think the user jobs will just hang, trying desperately to publish notices for the broker.

The post_retry logic (actually all retry logic) depends on having one retry list / process. Each instance has a one file per retry queue (download and post being extant currently.) In the context of libsr3shim... this does not make much sense. the processes are typically short-lived, non-daemons.
* There will be hundreds of thousands of post_retry.pid files created (1 per process.) if the broker goes down.
* The pid itself cannot retry the posts, because in order to keep user jobs going, the process has to end.

An alternative to the thousands of .pid files, would be to post to a pipe, or a named pipe, per node... in which case, you need a janitor that reads the named pipe. You end up creating a second IPC network to robustify your IPC network.

Taking the simpler option:

* we would create post_retry files per process... so there would be hundreds of thousands of such files created during a run. these processes end (they will not retry the posting themselves.)
* something need a (likely python scheduled watch?) janitor process that finds the post_retry.pid files and perhaps puts them into conventional retry_queue, and deletes the post_retry.pid files. To avoid contention, it makes sure the files it reads are > 1 minute old, before trying to process them.
* the janitor then needs to retry the posts conventionally.

This is one suggested implementation.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the post_retry and retry-list paths in libsr3shim, including how per-process .pid files are created when broker publishing fails. Clarify and implement the selected janitor and retry-queue design, including age and cleanup behavior, then verify that short-lived user jobs do not hang and failed posts are retried.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, python
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.