feat: add to_sedonadb() method
- Dominant language
- Java
- Stars
- 2.4k
- Forks
- 784
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 58
Description
It would be nice to have an interface that converts a SedonaSpark DataFrame to a SedonaDB DataFrame easily. Here is a current solution that works:
```python
import sedona.db
sd = sedona.db.connect()
df = sd.create_data_frame(dataframe_to_arrow(spark_df))
```
This could be nice:
```python
spark_df.to_sedonadb()
```
But maybe we'd have to do this:
```python
spark_df.to_sedonadb(sd)
```
This would allow for cool spatial workflows, like this:
* Read an Iceberg table with SedonaSpark and perform big data operations with a filtering operation at the end to make the data small enough to fit on a single machine
* Convert the SedonaSpark DataFrame to SedonaDB
* Use a library that's compatible with SedonaDB, like lonboard, to create a graph
Let me know what you think!
Contributor guide
Research direction
Start by reviewing the current conversion path shown in the issue: dataframe_to_arrow(spark_df), sedona.db.connect(), and create_data_frame(). Decide whether the interface should accept a SedonaDB connection or create one implicitly; done means a documented to_sedonadb() workflow supports the proposed SedonaSpark-to-SedonaDB conversion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, spark
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100