Databricks + DBT + Sedona compatibility

At our startup, we use the combination of Sedona and DBT on Databricks. However, the compatibility of latest Sedona versions with Databricks Runtime is not easy or clear to understand.

Following the documentation I though 1.7.2 and above should be compatible with DBR 16.4 LTS. I was able to run the Sedona spatial functions, but I could not the resulting dataset as Delta file. Using DBT I ran into a following error:

I have tried the following combinations:

- Sedona 1.7.2 + DBR 16.4 (above error)
- Sedona 1.8.0 + DBR 16.4 (above error)
- Sedona 1.8.0 + DBR 17.X (some Spark 4.0 incompatibility)

Does anyone here have experience with the setup? If there is something in development that we might test using our setup, I would be open to support that.

Until then, it seems that the Sedona 1.7.0 and DBR 15.4 is just the one stable solution.

Start by reproducing the reported Sedona 1.7.2 and 1.8.0 combinations with Databricks Runtime 16.4, checking the DBT error when writing the resulting dataset as Delta. Compare the compatibility documentation and test the reported DBR 17.X Spark 4.0 incompatibility. Done means identifying a supported combination or documenting the incompatibility and required change.

Written by the indexing model from the issue text.