airbytehq / airbytehq/PyAirbyte

Support for Prefect 3

Open
#365 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
344
Forks
77
Avg merge
1d 11h
Merged PRs (30d)
35

Description

### Topic

Make Airbyte compatible with Prefect 3

### Relevant information

Since we are blocked by [parallelism support](https://github.com/airbytehq/airbyte/issues/7750) we are running Prefect + Pyairbyte and want to upgrade to Prefect 3 but are blocked on the dependency of airbyte-cdk (which is a dependency of pyairbyte) on jsonschema = "~3.2.0" and pendulum = "<3.0.0".
[Prefect 3](https://docs.prefect.io/3.0/resources/whats-new-prefect-3) brings a lot of performance improvements to utilise pyairbyte to the fullest.
Moreover,
1. jsonschema 4.0.0 was released on Sep 30, 2021 and not upgrading does not make sense.
2. pendulum 3.0.0 brings lots of performance improvement with extensions rewritten in Rust

Upgrading these core dependencies ensures compatibility across various other integrations with airbyte.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.