Aiven-Open / Aiven-Open/metadata-parser

[EPIC] Apache Kafka Connect Connectors

Open
#10 0 comments 0 reactions 0 assignees View on GitHub
help wanted
Dominant language
Python
Stars
14
Forks
7
PR merge metrics
No merged PRs in 30d

Description

# What is currently missing?

The Apache Kafka Connect Connector parsing is tedious since we need to parse the JSON definition for each of them. As of now the basic Debezium PG source and OpenSearch sink are done but can be improved. Other connectors (see `explore_kafka_connect` in `explore_service.py`) are:

**Source**

* [ ] [Couchbase](https://github.com/couchbase/kafka-connect-couchbase)
* [ ] [Official MongoDB](https://docs.mongodb.com/kafka-connector/current/) preview
* [ ] [Debezium for MongoDB](https://debezium.io/docs/connectors/mongodb/)
* [ ] [Debezium for MySQL](https://debezium.io/docs/connectors/mysql/)
* [X] [Debezium for PostgreSQL](https://developer.aiven.io/docs/products/kafka/kafka-connect/howto/debezium-source-connector-pg.html)
* [ ] [Debezium for SQL Server](https://debezium.io/docs/connectors/sqlserver/)
* [ ] [Google Cloud Pub/Sub](https://github.com/GoogleCloudPlatform/pubsub/tree/master/kafka-connector)
* [ ] [Google Cloud Pub/Sub Lite](https://github.com/GoogleCloudPlatform/pubsub/)
* [ ] [JDBC](https://github.com/aiven/aiven-kafka-connect-jdbc/blob/master/docs/source-connector.md)
* [ ] Schema Source
* [ ] [Stream Reactor Cassandra](https://docs.lenses.io/connectors/source/cassandra.html)
* [ ] [Stream Reactor MQTT](https://docs.lenses.io/connectors/source/mqtt.html)

**Sink**

* [ ] [Aiven for Apache Kafka S3 Sink Connector](https://developer.aiven.io/docs/products/kafka/kafka-connect/howto/s3-sink-connector-aiven.html)
* [ ] [Confluent Amazon S3 Sink](https://help.aiven.io/kafka/aiven-kafka-kafka-connect-s3)
* [ ] [Couchbase](https://github.com/couchbase/kafka-connect-couchbase)
* [ ] [Elasticsearch](https://help.aiven.io/kafka/aiven-kafka-elasticsearch-sink-connector)
* [ ] [Google BigQuery](https://github.com/wepay/kafka-connect-bigquery)
* [ ] [Google Cloud Pub/Sub](https://github.com/GoogleCloudPlatform/pubsub/)
* [ ] [Google Cloud Pub/Sub Lite](https://github.com/GoogleCloudPlatform/pubsub/)
* [ ] [Google Cloud Storage](https://help.aiven.io/kafka/connectors/aiven-kafka-gcs-sink-connector)
* [ ] [HTTP](https://github.com/aiven/aiven-kafka-connect-http) preview
* [ ] [JDBC](https://github.com/aiven/aiven-kafka-connect-jdbc/blob/master/docs/sink-connector.md)
* [ ] [Official MongoDB](https://docs.mongodb.com/kafka-connector/current/) preview
* [X] [OpenSearch](https://github.com/aiven/opensearch-connector-for-apache-kafka/blob/main/docs/opensearch-sink-connector-config-options.rst) preview
* [ ] [Snowflake](https://docs.snowflake.net/manuals/user-guide/kafka-connector.html) preview
* [ ] [Splunk](https://github.com/splunk/kafka-connect-splunk)
* [ ] [Stream Reactor Cassandra](https://docs.lenses.io/connectors/sink/cassandra.html)
* [ ] [Stream Reactor InfluxDB](https://docs.lenses.io/connectors/sink/influx.html)
* [ ] [Stream Reactor Mongo DB](https://docs.lenses.io/connectors/sink/mongo.html)
* [ ] [Stream Reactor MQTT](https://docs.lenses.io/connectors/sink/mqtt.html)
* [ ] [Stream Reactor Redis](https://docs.lenses.io/connectors/sink/redis.html)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.