airbytehq / airbytehq/PyAirbyte

💡 Feature Request: Ability to use database-type sources, such as `source-postgres` and `source-mysql`

Open
#87 7 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
344
Forks
77
Avg merge
1d 11h
Merged PRs (30d)
35

Description

Because most database sources are built on Java, they currently are not able to run in Python environments.

This feature would allow database-type sources to be run from PyAirbyte. Possible implementation options could be:

1. Allow docker containers to be invoked by Airbyte.
- **Important:** While this _technically_ is not a big lift, not sure if we want to take this approach - as it would create a very sharp difference in user experience for those runtimes which have docker access, versus those which do not.
2. Find a way to package and install Java connectors as standalone executables.
- This is in theory _also_ technically feasible, but this approach is subject to its own sharp edges - such as needing to have pre-built Java executables for `n` number of platforms/runtimes.

## Available Workarounds

### Workaround # 1: Pre-Installing the Java-based connector

One workaround is to pre-install the Java-based connector on your local machine or docker image, and then create a CLI which can mimic the CLI of a Python-based connector. If registered on PATH, PyAirbyte will find this connector and not know/care what language it is written in.

### Workaround # 2: Treating the source DB as an externally-managed "cache"

An alternative workaround, which admittedly would not solve all use cases, would be what is described in this issue:

- #85

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.