agile-lab-dev / agile-lab-dev/darwin

Kafka connector

Offen
#66 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
contest good first issue
Vorherrschende Sprache
Scala
Sterne
36
Forks
10
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

Do you think that make sense to have a Kafka connector? Store schemas directly in Kafka?

We can have a topic with infinite retention, where the key is the schema id and the value is the schema (plus other info if?).
This connector can write to this topic and maintain a list of id->schema in memory by consuming its own the records.
Of course the assumption is that we don't have milions of schemas...

Each instance of this connector reads and maintain all the schemas in memory, so each instance is indipendent from the other.
At every startup it reads the topic from the start.

The main benefit is that we don't have external dependencies for use cases where we just use Kafka and that we receive new schemas immediately.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.