locationtech / locationtech/geomesa
Add Python bindings for JTS + GeoMesa Spark UDFs
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 1.5k
- Forks
- 446
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 31
Description
Add Python bindings for JTS + GeoMesa Spark UDFs
RasterFrames has examples of how to register UDFs with the PySpark context.
There may be other Python packaging tricks that the RasterFrames team has sorted out.
Original JIRA Issue: https://geomesa.atlassian.net/browse/GEOMESA-2766
Key: GEOMESA-2766
Type: Improvement
Priority: Major
Status: To Do
Resolution: Unresolved
Reporter: James Hughes
Created: Fri, 10 Jan 2020 10:29:20 -0500
Updated: Fri, 10 Jan 2020 10:35:11 -0500
Comments:
557058:f542d9ac-6efc-4b69-93fb-160ff09c0414 (Fri, 10 Jan 2020 10:31:58 -0500):
This is the easy part. Creating a self-contained Python .whl file that navigates the nine levels of pip hell is the hard part.
557058:878133e6-01b0-4022-8f67-5e77d62963f5 (Fri, 10 Jan 2020 10:35:11 -0500):
Of interest from the RasterFrames side of the house is
- Making sure any SparkSession / SparkContext modifications made by either package are copacetic. Bonus points if we can also include geopyspark in this mix. Extra bonus points for documenting useful patterns
- Packaging / distribution as Simeon Fitch points out. Currently RasterFrames is oriented toward producing a publishable whl. But GeoMesa is not
- Separation of concerns. Defining GeoMesa python functions and python UDTs within RasterFrames is madness.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing RasterFrames' pyrasterframes/rasterfunctions.py, especially lines 718-1062, and its PySpark UDF registration approach. Compare that with GeoMesa's Spark UDF and Python packaging structure; done means Python bindings are distributable, coexist cleanly with SparkSession or SparkContext changes, and include the useful usage patterns described in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100