locationtech / locationtech/geomesa

SparkSQL - support schemas without geometries

Open
#6,460 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Scala
Stars
1.5k
Forks
446
Avg merge
1d 8h
Merged PRs (30d)
31

Description

SparkSQL - support schemas without geometries

Trying to save a schema without a geometry field throws an NPE.

It looks like the issue is from [this|https://github.com/locationtech/geomesa/blob/geomesa_2.11-2.3.0/geomesa-spark/geomesa-spark-sql/src/main/scala/org/locationtech/geomesa/spark/GeoMesaSparkSQL.scala#L240
] geom index line. We should be able to make the variable lazy, or add an Option.map.getOrElse(-1) so that it doesn't throw an NPE if the geometry descriptor is null. (Rémi Grisot verified making it lazy worked)

I only see that variable being used in two places - one is guarded by an 'isSpatial' check, and the other is only triggered if the user passes in 'spatial' as an argument, so it's probably safe to assume there will be a valid geom in those places.


Original JIRA Issue: https://geomesa.atlassian.net/browse/GEOMESA-2629

Key: GEOMESA-2629
Type: Improvement
Priority: Major
Status: To Do
Resolution: Unresolved
Reporter: Rémi Grisot
Created: Thu, 23 May 2019 08:37:28 -0400
Updated: Tue, 14 Apr 2020 15:53:44 -0400


Comments:

557058:48912df4-0004-4f90-9915-db26684ec006 (Tue, 14 Apr 2020 15:40:02 -0400):

Rémi Grisot we’ve usually taken the position that GeoMesa is aimed at indexing geospatial data. Part of that decision involves using the GeoTools DataStore API and using SimpleFeatures as an underlying concept. SimpleFeatures always have a geometry, so that’s one reason that this issue is happening.

As such, we may close this ticket as “Won’t Fix” at some point.

557058:da7f6dfa-f0a1-468b-aec5-2cb61273feb5 (Tue, 14 Apr 2020 15:53:44 -0400):

Note that simple features don't have to have a geometry, but our support for schemas without geometries is spotty as it's not a primary use case.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in geomesa-spark/geomesa-spark-sql/src/main/scala/org/locationtech/geomesa/spark/GeoMesaSparkSQL.scala at the geom index initialization around line 240. Check how that value is used for non-spatial schemas and confirm that spatial paths still require a valid geometry. Done means saving a schema without a geometry no longer throws an NPE, while spatial behavior remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.