apache / apache/sedona

feat: add to_sedonadb() method

Open
#2,511 1 comment 0 reactions 0 assignees View on GitHub
good first issue help wanted
Dominant language
Java
Stars
2.4k
Forks
784
Avg merge
1d 12h
Merged PRs (30d)
58

Description

It would be nice to have an interface that converts a SedonaSpark DataFrame to a SedonaDB DataFrame easily. Here is a current solution that works:

```python
import sedona.db
sd = sedona.db.connect()

df = sd.create_data_frame(dataframe_to_arrow(spark_df))
```

This could be nice:

```python
spark_df.to_sedonadb()
```

But maybe we'd have to do this:

```python
spark_df.to_sedonadb(sd)
```

This would allow for cool spatial workflows, like this:

* Read an Iceberg table with SedonaSpark and perform big data operations with a filtering operation at the end to make the data small enough to fit on a single machine
* Convert the SedonaSpark DataFrame to SedonaDB
* Use a library that's compatible with SedonaDB, like lonboard, to create a graph

Let me know what you think!

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the current conversion path shown in the issue: dataframe_to_arrow(spark_df), sedona.db.connect(), and create_data_frame(). Decide whether the interface should accept a SedonaDB connection or create one implicitly; done means a documented to_sedonadb() workflow supports the proposed SedonaSpark-to-SedonaDB conversion.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, spark
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.