Databricks Spark Configuration in Documentation
- Dominant language
- Java
- Stars
- 2.4k
- Forks
- 784
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 58
Description
In the Databricks Spark documentation here: https://sedona.apache.org/1.3.1-incubating/setup/databricks/
It says recommends the following Spark configuration:
```
spark.sql.extensions org.apache.sedona.viz.sql.SedonaVizExtensions,org.apache.sedona.sql.SedonaSqlExtensions
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.kryo.registrator org.apache.sedona.core.serde.SedonaKryoRegistrator
```
However, after speaking with Carto support, they said that the only configuration necessary was this:
```
spark.sql.extensions com.carto.analytics.toolbox.sql.SparkExtension
spark.databricks.geo.st.enabled true
```
Granted this includes a Carto analytics toolbox, and I don't know what is in that, but does the spark.databricks.geo.st.enabled true handle the KryoSerializer and KryoRegistrator now? With this configuration listed above, the KryoSerializer, KryoRegistrator were not needed. SedonaSqlExtensions and SedonaVizExtensions were also not needed. I just wanted to flag this as something may have changed on the Databricks side obviating the need for these configurations, replacing them with spark.databricks.geo.st.enabled = true.
Contributor guide
Research direction
Start with the Databricks setup page at https://sedona.apache.org/1.3.1-incubating/setup/databricks/ and compare its Spark configuration with the Carto configuration quoted in the issue. Verify whether spark.databricks.geo.st.enabled replaces the listed extensions and Kryo settings, then update the documentation to describe the currently supported setup and any required caveats.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- spark
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100