airbytehq / airbytehq/PyAirbyte

Docs or Sample Script: Re-usable example to launch Java connectors from PyAirbyte, without docker

Abierto
#700 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
344
Forks
77
Merge medio
1 d 11 h
PR fusionados (30 d)
35

Descripción

This comes up frequently. It's _theoretically_ completely possible for PyAirbyte to run Java connectors natively - because all PyAirbyte cares about is having a CLI it can invoke.

Preview discussion has happened here:

- #87

## Discovery Work (Spelunking)

I've recently been digging into the internals of how Java docker images are built - and I learned about the `airbyte/base.sh` shell script which the Java connector Dockerfile uses as its container `ENTRYPOINT` script.

https://github.com/airbytehq/airbyte/blob/79f57cf7f441aef2dc959ae05cdb38a706079702/docker-images/Dockerfile.java-connector#L14-L17

The logic for that script is here:

https://github.com/airbytehq/airbyte/blob/79f57cf7f441aef2dc959ae05cdb38a706079702/airbyte-integrations/bases/base/base.sh

```bash
case "$CMD" in
spec)
eval "$AIRBYTE_SPEC_CMD"
;;
check)
eval "$AIRBYTE_CHECK_CMD" --config "$CONFIG_FILE"
;;
discover)
eval "$AIRBYTE_DISCOVER_CMD" --config "$CONFIG_FILE"
;;
read)
READ_STATEMENT="$AIRBYTE_READ_CMD --config $CONFIG_FILE --catalog $CATALOG_FILE"
if [[ ! -z "$STATE_FILE" ]]; then READ_STATEMENT="$READ_STATEMENT --state $STATE_FILE"; fi
eval "$READ_STATEMENT"
;;
write)
eval "$AIRBYTE_WRITE_CMD" --config "$CONFIG_FILE" --catalog "$CATALOG_FILE"
;;
*)
error "Unknown command: $CMD"
;;
esac
```

And the `AIRBYTE_*_CMD` env vars are declared in the base docker file:

https://github.com/airbytehq/airbyte/blob/79f57cf7f441aef2dc959ae05cdb38a706079702/docker-images/Dockerfile.java-connector-base

```bash
ENV AIRBYTE_SPEC_CMD="/airbyte/javabase.sh --spec"
ENV AIRBYTE_CHECK_CMD="/airbyte/javabase.sh --check"
ENV AIRBYTE_DISCOVER_CMD="/airbyte/javabase.sh --discover"
ENV AIRBYTE_READ_CMD="/airbyte/javabase.sh --read"
ENV AIRBYTE_WRITE_CMD="/airbyte/javabase.sh --write"
ENV AIRBYTE_ENTRYPOINT="/airbyte/base.sh"
```

And `javabase.sh` is defined here:

https://github.com/airbytehq/airbyte/blob/79f57cf7f441aef2dc959ae05cdb38a706079702/airbyte-integrations/bases/base-java/javabase.sh

Essentially:

```bash
JAVA_OPTS=$JAVA_OPTS" -Dlog4j.encoder.byteBufferSize=32768 -Dlog4j2.configurationFile=log4j2.xml"
#needed because we make ThreadLocal.get(Thread) accessible in IntegrationRunner.stopOrphanedThreads
JAVA_OPTS=$JAVA_OPTS" --add-opens=java.base/java.lang=ALL-UNNAMED"
# tell jooq to be quiet (https://stackoverflow.com/questions/28272284/how-to-disable-jooqs-self-ad-message-in-3-4)
JAVA_OPTS=$JAVA_OPTS" -Dorg.jooq.no-logo=true -Dorg.jooq.no-tips=true"
export JAVA_OPTS

#...
if [[ $A = --write ]]; then
cat <&0 | /airbyte/bin/"$APPLICATION" "$@"
else
/airbyte/bin/"$APPLICATION" "$@"
fi
```

with `/airbyte/bin/"$APPLICATION"` referring back to the declaration in the connector Dockerfile:

https://github.com/airbytehq/airbyte/blob/79f57cf7f441aef2dc959ae05cdb38a706079702/docker-images/Dockerfile.java-connector#L18C1-L18C36

```bash
ENV APPLICATION="${CONNECTOR_NAME}"
```

## Possible Plan of Attach

1. Java is pre-install in their container, or there is a compatible JRE/JDK bundled or downloaded on demand.
2. We run something like `./gradlew :airbyte-integrations:connectors:source-mssql:distTar` to build the tarball for the connector.
3. We deliver the tarball to the PyAirbyte runtime, extracting its contents as the java container image would: https://github.com/airbytehq/airbyte/blob/79f57cf7f441aef2dc959ae05cdb38a706079702/docker-images/Dockerfile.java-connector#L20-L30
4. We invoke the java connector, following the pattern in `javabase.sh` above.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.