airbytehq / airbytehq/PyAirbyte
Docs or Sample Script: Re-usable example to launch Java connectors from PyAirbyte, without docker
- Langage dominant
- Python
- Étoiles
- 344
- Forks
- 77
- Merge moyen
- 1 j 11 h
- PR mergées (30 j)
- 35
Description
This comes up frequently. It's _theoretically_ completely possible for PyAirbyte to run Java connectors natively - because all PyAirbyte cares about is having a CLI it can invoke.
Preview discussion has happened here:
- #87
## Discovery Work (Spelunking)
I've recently been digging into the internals of how Java docker images are built - and I learned about the `airbyte/base.sh` shell script which the Java connector Dockerfile uses as its container `ENTRYPOINT` script.
https://github.com/airbytehq/airbyte/blob/79f57cf7f441aef2dc959ae05cdb38a706079702/docker-images/Dockerfile.java-connector#L14-L17
The logic for that script is here:
https://github.com/airbytehq/airbyte/blob/79f57cf7f441aef2dc959ae05cdb38a706079702/airbyte-integrations/bases/base/base.sh
```bash
case "$CMD" in
spec)
eval "$AIRBYTE_SPEC_CMD"
;;
check)
eval "$AIRBYTE_CHECK_CMD" --config "$CONFIG_FILE"
;;
discover)
eval "$AIRBYTE_DISCOVER_CMD" --config "$CONFIG_FILE"
;;
read)
READ_STATEMENT="$AIRBYTE_READ_CMD --config $CONFIG_FILE --catalog $CATALOG_FILE"
if [[ ! -z "$STATE_FILE" ]]; then READ_STATEMENT="$READ_STATEMENT --state $STATE_FILE"; fi
eval "$READ_STATEMENT"
;;
write)
eval "$AIRBYTE_WRITE_CMD" --config "$CONFIG_FILE" --catalog "$CATALOG_FILE"
;;
*)
error "Unknown command: $CMD"
;;
esac
```
And the `AIRBYTE_*_CMD` env vars are declared in the base docker file:
https://github.com/airbytehq/airbyte/blob/79f57cf7f441aef2dc959ae05cdb38a706079702/docker-images/Dockerfile.java-connector-base
```bash
ENV AIRBYTE_SPEC_CMD="/airbyte/javabase.sh --spec"
ENV AIRBYTE_CHECK_CMD="/airbyte/javabase.sh --check"
ENV AIRBYTE_DISCOVER_CMD="/airbyte/javabase.sh --discover"
ENV AIRBYTE_READ_CMD="/airbyte/javabase.sh --read"
ENV AIRBYTE_WRITE_CMD="/airbyte/javabase.sh --write"
ENV AIRBYTE_ENTRYPOINT="/airbyte/base.sh"
```
And `javabase.sh` is defined here:
https://github.com/airbytehq/airbyte/blob/79f57cf7f441aef2dc959ae05cdb38a706079702/airbyte-integrations/bases/base-java/javabase.sh
Essentially:
```bash
JAVA_OPTS=$JAVA_OPTS" -Dlog4j.encoder.byteBufferSize=32768 -Dlog4j2.configurationFile=log4j2.xml"
#needed because we make ThreadLocal.get(Thread) accessible in IntegrationRunner.stopOrphanedThreads
JAVA_OPTS=$JAVA_OPTS" --add-opens=java.base/java.lang=ALL-UNNAMED"
# tell jooq to be quiet (https://stackoverflow.com/questions/28272284/how-to-disable-jooqs-self-ad-message-in-3-4)
JAVA_OPTS=$JAVA_OPTS" -Dorg.jooq.no-logo=true -Dorg.jooq.no-tips=true"
export JAVA_OPTS
#...
if [[ $A = --write ]]; then
cat <&0 | /airbyte/bin/"$APPLICATION" "$@"
else
/airbyte/bin/"$APPLICATION" "$@"
fi
```
with `/airbyte/bin/"$APPLICATION"` referring back to the declaration in the connector Dockerfile:
https://github.com/airbytehq/airbyte/blob/79f57cf7f441aef2dc959ae05cdb38a706079702/docker-images/Dockerfile.java-connector#L18C1-L18C36
```bash
ENV APPLICATION="${CONNECTOR_NAME}"
```
## Possible Plan of Attach
1. Java is pre-install in their container, or there is a compatible JRE/JDK bundled or downloaded on demand.
2. We run something like `./gradlew :airbyte-integrations:connectors:source-mssql:distTar` to build the tarball for the connector.
3. We deliver the tarball to the PyAirbyte runtime, extracting its contents as the java container image would: https://github.com/airbytehq/airbyte/blob/79f57cf7f441aef2dc959ae05cdb38a706079702/docker-images/Dockerfile.java-connector#L20-L30
4. We invoke the java connector, following the pattern in `javabase.sh` above.
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.