SparkSQL - support writing DecimalTypes and BigInts
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 65/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Quiet
- Domain
- data-engineering
Research direction
Start in GeoMesaSparkSQL.scala at structType2SFT, especially the handling reported around lines 115-117, and trace how createRelation converts the Spark SQL schema for writes. Add support for DecimalType and BigInt values, then verify that the SparkSQL GeoMesa write shown in the issue completes successfully.
Written by the indexing model from the issue text.
Description
SparkSQL - support writing DecimalTypes and BigInts
geodf.write.format("geomesa").option("fs.path", "myGeomesaFSDS").option("geomesa.feature", "featureName").save cala.MatchError: DecimalType(38,0) (of class org.apache.spark.sql.types.DecimalType) at org.locationtech.geomesa.spark.GeoMesaDataSource$$anonfun$structType2SFT$2.apply(GeoMesaSparkSQL.scala:117) at org.locationtech.geomesa.spark.GeoMesaDataSource$$anonfun$structType2SFT$2.apply(GeoMesaSparkSQL.scala:116) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at org.locationtech.geomesa.spark.GeoMesaDataSource.structType2SFT(GeoMesaSparkSQL.scala:115) at org.locationtech.geomesa.spark.GeoMesaDataSource.createRelation(GeoMesaSparkSQL.scala:166) at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:471)
Original JIRA Issue: https://geomesa.atlassian.net/browse/GEOMESA-2285
Key: GEOMESA-2285
Type: Story
Priority: Major
Status: To Do
Resolution: Unresolved
Reporter: Emilio Lahr-Vivaz
Created: Fri, 25 May 2018 11:05:18 -0400
Updated: Fri, 1 Jun 2018 08:37:09 -0400
- Dominant language
- Scala
- Stars
- 1.5k
- Forks
- 446
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 31
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from locationtech/geomesa
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
locationtech/geomesa#10761 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
locationtech/geomesa#10324 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
locationtech/geomesa#10111 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
locationtech/geomesa#10052 ·
-
Difficulty 1/5 1-3 hours Newbie friendliness 78/100
locationtech/geomesa#9998 ·
All issues in locationtech/geomesa
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
-
requires-triage
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
apache/datafusion-comet#6060 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
lichess-org/lila#21756 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
salesforce/evalon#10 ·