locationtech / locationtech/geomesa

Add Python bindings for JTS + GeoMesa Spark UDFs

Open
#6,596 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Scala
Stars
1.5k
Forks
446
Avg merge
1d 8h
Merged PRs (30d)
31

Description

Add Python bindings for JTS + GeoMesa Spark UDFs

RasterFrames has examples of how to register UDFs with the PySpark context.

https://github.com/locationtech/rasterframes/blob/43bd3b37f0b2931470b2d5b5757474d3c885e659/pyrasterframes/src/main/python/pyrasterframes/rasterfunctions.py#L718-L1062

There may be other Python packaging tricks that the RasterFrames team has sorted out.


Original JIRA Issue: https://geomesa.atlassian.net/browse/GEOMESA-2766

Key: GEOMESA-2766
Type: Improvement
Priority: Major
Status: To Do
Resolution: Unresolved
Reporter: James Hughes
Created: Fri, 10 Jan 2020 10:29:20 -0500
Updated: Fri, 10 Jan 2020 10:35:11 -0500


Comments:

557058:f542d9ac-6efc-4b69-93fb-160ff09c0414 (Fri, 10 Jan 2020 10:31:58 -0500):

This is the easy part. Creating a self-contained Python .whl file that navigates the nine levels of pip hell is the hard part.

557058:878133e6-01b0-4022-8f67-5e77d62963f5 (Fri, 10 Jan 2020 10:35:11 -0500):

Of interest from the RasterFrames side of the house is

  • Making sure any SparkSession / SparkContext modifications made by either package are copacetic. Bonus points if we can also include geopyspark in this mix. Extra bonus points for documenting useful patterns
  • Packaging / distribution as Simeon Fitch points out. Currently RasterFrames is oriented toward producing a publishable whl. But GeoMesa is not
  • Separation of concerns. Defining GeoMesa python functions and python UDTs within RasterFrames is madness.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing RasterFrames' pyrasterframes/rasterfunctions.py, especially lines 718-1062, and its PySpark UDF registration approach. Compare that with GeoMesa's Spark UDF and Python packaging structure; done means Python bindings are distributable, coexist cleanly with SparkSession or SparkContext changes, and include the useful usage patterns described in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, spark
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.