airbytehq / airbytehq/PyAirbyte

Docs or Sample Script: Re-usable example to launch Java connectors from PyAirbyte, without docker

Đang mở
#700 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
344
Fork
77
Merge trung bình
1 ngày 11 giờ
Pull request đã merge (30 ngày)
35

Mô tả

This comes up frequently. It's _theoretically_ completely possible for PyAirbyte to run Java connectors natively - because all PyAirbyte cares about is having a CLI it can invoke.

Preview discussion has happened here:

- #87

## Discovery Work (Spelunking)

I've recently been digging into the internals of how Java docker images are built - and I learned about the `airbyte/base.sh` shell script which the Java connector Dockerfile uses as its container `ENTRYPOINT` script.

https://github.com/airbytehq/airbyte/blob/79f57cf7f441aef2dc959ae05cdb38a706079702/docker-images/Dockerfile.java-connector#L14-L17

The logic for that script is here:

https://github.com/airbytehq/airbyte/blob/79f57cf7f441aef2dc959ae05cdb38a706079702/airbyte-integrations/bases/base/base.sh

```bash
case "$CMD" in
spec)
eval "$AIRBYTE_SPEC_CMD"
;;
check)
eval "$AIRBYTE_CHECK_CMD" --config "$CONFIG_FILE"
;;
discover)
eval "$AIRBYTE_DISCOVER_CMD" --config "$CONFIG_FILE"
;;
read)
READ_STATEMENT="$AIRBYTE_READ_CMD --config $CONFIG_FILE --catalog $CATALOG_FILE"
if [[ ! -z "$STATE_FILE" ]]; then READ_STATEMENT="$READ_STATEMENT --state $STATE_FILE"; fi
eval "$READ_STATEMENT"
;;
write)
eval "$AIRBYTE_WRITE_CMD" --config "$CONFIG_FILE" --catalog "$CATALOG_FILE"
;;
*)
error "Unknown command: $CMD"
;;
esac
```

And the `AIRBYTE_*_CMD` env vars are declared in the base docker file:

https://github.com/airbytehq/airbyte/blob/79f57cf7f441aef2dc959ae05cdb38a706079702/docker-images/Dockerfile.java-connector-base

```bash
ENV AIRBYTE_SPEC_CMD="/airbyte/javabase.sh --spec"
ENV AIRBYTE_CHECK_CMD="/airbyte/javabase.sh --check"
ENV AIRBYTE_DISCOVER_CMD="/airbyte/javabase.sh --discover"
ENV AIRBYTE_READ_CMD="/airbyte/javabase.sh --read"
ENV AIRBYTE_WRITE_CMD="/airbyte/javabase.sh --write"
ENV AIRBYTE_ENTRYPOINT="/airbyte/base.sh"
```

And `javabase.sh` is defined here:

https://github.com/airbytehq/airbyte/blob/79f57cf7f441aef2dc959ae05cdb38a706079702/airbyte-integrations/bases/base-java/javabase.sh

Essentially:

```bash
JAVA_OPTS=$JAVA_OPTS" -Dlog4j.encoder.byteBufferSize=32768 -Dlog4j2.configurationFile=log4j2.xml"
#needed because we make ThreadLocal.get(Thread) accessible in IntegrationRunner.stopOrphanedThreads
JAVA_OPTS=$JAVA_OPTS" --add-opens=java.base/java.lang=ALL-UNNAMED"
# tell jooq to be quiet (https://stackoverflow.com/questions/28272284/how-to-disable-jooqs-self-ad-message-in-3-4)
JAVA_OPTS=$JAVA_OPTS" -Dorg.jooq.no-logo=true -Dorg.jooq.no-tips=true"
export JAVA_OPTS

#...
if [[ $A = --write ]]; then
cat <&0 | /airbyte/bin/"$APPLICATION" "$@"
else
/airbyte/bin/"$APPLICATION" "$@"
fi
```

with `/airbyte/bin/"$APPLICATION"` referring back to the declaration in the connector Dockerfile:

https://github.com/airbytehq/airbyte/blob/79f57cf7f441aef2dc959ae05cdb38a706079702/docker-images/Dockerfile.java-connector#L18C1-L18C36

```bash
ENV APPLICATION="${CONNECTOR_NAME}"
```

## Possible Plan of Attach

1. Java is pre-install in their container, or there is a compatible JRE/JDK bundled or downloaded on demand.
2. We run something like `./gradlew :airbyte-integrations:connectors:source-mssql:distTar` to build the tarball for the connector.
3. We deliver the tarball to the PyAirbyte runtime, extracting its contents as the java container image would: https://github.com/airbytehq/airbyte/blob/79f57cf7f441aef2dc959ae05cdb38a706079702/docker-images/Dockerfile.java-connector#L20-L30
4. We invoke the java connector, following the pattern in `javabase.sh` above.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.