[FEATURE REQ] Contribution for Spark 4.0 OLTP Connector for Cosmos DB

I am planning to upgrade our production data landscape (Databricks) to DBR 17.3 LTS, which runs on Apache Spark 4.0. Currently, the azure-cosmos-spark connector only supports up to Spark 3.5.

I am aware of issue #46290 stating a planned release for Q1 2026. However, due to our internal upgrade timelines, we need this support sooner.

Proposal I am a Senior Data Engineer with extensive Scala/Spark experience and I am willing to contribute the development of the Spark 4.0 module to this repository.

My proposed plan:

Create a new module sdk/cosmos/azure-cosmos-spark_4-0_2-13.

Port the existing 3.5 logic to support Scala 2.13 (required by Spark 4.0).

Address breaking changes in the Spark 4.0 DataSource V2 API.

Question for Maintainers (@kushagraThapar / @TheovanKraay)

Has internal work already started on this?

If yes, is there a feature branch I can contribute to?

If no, are you open to accepting a PR for this new module if I follow the contribution guidelines?

Setup

OS: Windows 11

IDE: IntelliJ / VS Code

Build: Maven 3.8+ / Java 17