apache / apache/sedona

Databricks Spark Configuration in Documentation

Open
#1,792 0 comments 0 reactions 0 assignees View on GitHub
docs
Dominant language
Java
Stars
2.4k
Forks
784
Avg merge
1d 12h
Merged PRs (30d)
58

Description

In the Databricks Spark documentation here: https://sedona.apache.org/1.3.1-incubating/setup/databricks/

It says recommends the following Spark configuration:
```
spark.sql.extensions org.apache.sedona.viz.sql.SedonaVizExtensions,org.apache.sedona.sql.SedonaSqlExtensions
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.kryo.registrator org.apache.sedona.core.serde.SedonaKryoRegistrator
```

However, after speaking with Carto support, they said that the only configuration necessary was this:
```
spark.sql.extensions com.carto.analytics.toolbox.sql.SparkExtension
spark.databricks.geo.st.enabled true
```

Granted this includes a Carto analytics toolbox, and I don't know what is in that, but does the spark.databricks.geo.st.enabled true handle the KryoSerializer and KryoRegistrator now? With this configuration listed above, the KryoSerializer, KryoRegistrator were not needed. SedonaSqlExtensions and SedonaVizExtensions were also not needed. I just wanted to flag this as something may have changed on the Databricks side obviating the need for these configurations, replacing them with spark.databricks.geo.st.enabled = true.

Contributor guide

Open the contributing guide

Research direction

Start with the Databricks setup page at https://sedona.apache.org/1.3.1-incubating/setup/databricks/ and compare its Spark configuration with the Carto configuration quoted in the issue. Verify whether spark.databricks.geo.st.enabled replaces the listed extensions and Kryo settings, then update the documentation to describe the currently supported setup and any required caveats.

Written by the indexing model from the issue text.

Assessment

Tech stack
spark
Domain
documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.