fluent / fluent/fluent-logger-python

Support `chunk` option to enable "at-least-once" delivery

Ouverte
#120 3 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Python
Étoiles
457
Forks
138
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

### Background

According to the "Forward Protocol Specification v1", fluentd supports an option named `chunk` which enables at-least-once delivery of messages.

This option is very useful in cases where data loss is not acceptable.

> https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1#option

### The problem

The current design of fluent-logger-python, however, makes it difficult to support this new option.
Specifically:

1. Events are buffered inside `FluentSender` class as a single `bytes` sequence (`self.pendings`). There is no efficient way to reconstruct a specific event from the buffer and resend it.
2. And this `bytes` sequence buffer is kinda API. So we cannot moddify the format in which `FluentSender` buffers messages (at least, casually) or it will break many user-defined `buffer_overflow_handler`s.
3. Also for now, we lack a handful of building blocks for supporting the "at-least-once" semantics. For example, there is no reliable mechanism for users to tell if a message has been delivered successfully [^]

### So we need to ...

The bottom line is, we need to apply some architectural changes to make this library support the (newly-introduced) "at-least-once" semantics. Of course, we need to do it without breaking many existing programs.

What do you think about this? Or is there already a plan to make this library compliant with the v1 specification?

----

[^] Yes, `FluentSender.emit()` is supposed to notify this via its return value. But even if the method returns False, the message might be delivered anyway through the pending buffer, and this "retry" part is totally opaque to users.

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Piste de recherche

Start by reading the FluentSender implementation, especially its pendings buffer and emit() return-value behavior, then compare them with the Forward Protocol Specification v1 option section. Define how chunk support, retries, delivery results, and existing buffer_overflow_handler compatibility should work; done means the design and compatibility boundaries are agreed before implementation.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
python
Domaine
distributed-systems
Type d'issue
Fonctionnalité
Difficulté
5/5
Temps estimé
Plus d'une semaine
Activité
À l'abandon
Clarté
À clarifier
Accessibilité débutants
25/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.