sld heatmaps has issue when used with random set of linestrings for geomesa-hbase

sld heatmaps has issue when used with random set of linestrings for geomesa-hbase

I have a stable system and am mainly using heatmaps sld (https://www.geomesa.org/documentation/3.0.0/user/hbase/heatmaps.html?highlight=heatmaps#install-heatmap-sld ) for > 1.5 years for data where the geometric object that I store are points.
Currently it is important for me to also store data as Linestrings and apply heatmaps to them. I see this causes the heatmaps to fail with following error:
Caused by: java.lang.RuntimeException: Failed to evaluate the process function, error is: Error processing heatmap at org.geotools.process.function.ProcessFunction.evaluate(ProcessFunction.java:162) at org.geotools.renderer.lite.RenderingTransformationHelper.applyRenderingTransformation(RenderingTransformationHelper.java:291) at org.geotools.renderer.lite.StreamingRenderer.getFeatures(StreamingRenderer.java:2296) at org.geotools.renderer.lite.StreamingRenderer.processStylers(StreamingRenderer.java:2213) at org.geotools.renderer.lite.StreamingRenderer.paint(StreamingRenderer.java:900) at org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:601) ... 135 more Caused by: org.geotools.process.ProcessException: Error processing heatmap at org.locationtech.geomesa.process.analytic.DensityProcess.execute(DensityProcess.scala:75) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.geotools.process.factory.AnnotationDrivenProcessFactory$InvokeMethodProcess.execute(AnnotationDrivenProcessFactory.java:621) at org.geotools.process.function.ProcessFunction.evaluate(ProcessFunction.java:148) ... 140 more Caused by: java.lang.ArrayIndexOutOfBoundsException: 888 at org.geotools.process.vector.HeatmapSurface.addPoint(HeatmapSurface.java:100) at org.locationtech.geomesa.process.analytic.DensityProcess$$anonfun$execute$1.apply(DensityProcess.scala:81) at org.locationtech.geomesa.process.analytic.DensityProcess$$anonfun$execute$1.apply(DensityProcess.scala:76) at org.locationtech.geomesa.utils.io.package$WithClose$.apply(package.scala:64) at org.locationtech.geomesa.process.analytic.DensityProcess.execute(DensityProcess.scala:76)
I see that if I look at my dataset (typically about 30k lines) that the heatmap works for "some lines"-I can filter out lines and if I only have a few remaining it will work. Like this I can detect which lines cause the heatmaps to fail. But if I look at the "problematic" line that triggers the error, it is displayed correctly in geoserver and all coordinates look valid.
Further I notice that if I play with the radiusPixels parameter and put it to a very high number (which will cause the 1 linestring to be spread out over half of the globe), sometimes the error goes away and the heatmap is produced.

Things checked so far:
-believe distributed-runtime jar matches the same version as your geoserver (it has worked for millions of points with this setup)
-wrote a unit test in accumuo for a "problematic linestring", and this test doesnt fail (https://github.com/JB-data/geomesa-heatmap4lines/blob/main/geomesa-accumulo/geomesa-accumulo-datastore/src/test/scala/org/locationtech/geomesa/process/analytic/DensityProcessTest.scala)

Here an example of a problematic linestring of length 70:
LINESTRING (121.78333282470703 31.133333206176758, 121.7763900756836 31.140277862548828, 121.76249694824219 31.153888702392578, 121.7558364868164 31.16083335876465, 121.741943359375 31.1744441986084, 121.73500061035156 31.18138885498047, 121.71416473388672 31.20194435119629, 121.67972564697266 31.23611068725586, 121.63833618164062 31.2772216796875, 121.58999633789062 31.325000762939453, 121.45166778564453 31.462223052978516, 121.3758316040039 31.537500381469727, 121.28583526611328 31.626388549804688, 121.18916320800781 31.721389770507812, 121.08555603027344 31.822500228881836, 120.96138763427734 31.943889617919922, 120.81610870361328 32.08583450317383, 120.73332977294922 32.16666793823242, 120.6066665649414 32.176944732666016, 120.36360931396484 32.1966667175293, 120.09972381591797 32.21833419799805, 119.81444549560547 32.24166488647461, 119.76166534423828 32.24583435058594, 119.64305877685547 32.44722366333008, 119.46277618408203 32.75361251831055, 119.44833374023438 32.77777862548828, 117.21305847167969 36.41777801513672, 116.81639099121094 37.223331451416016, 116.37860870361328 38.11305618286133, 116.42500305175781 38.372501373291016, 116.57166290283203 39.19499969482422, 116.85444641113281 39.79944610595703, 116.76083374023438 40.02305603027344, 116.74305725097656 40.06611251831055, 115.95194244384766 41.95861053466797, 115.26277923583984 42.23611068725586, 105.59610748291016 41.20500183105469, 104.47333526611328 41.030555725097656, 95.19944763183594 41.86166763305664, 86.71611022949219 43.5433349609375, 84.3647232055664 43.34583282470703, 80.7330551147461 43.50027847290039, 79.8066635131836 43.53305435180664, 79.5594482421875 43.540279388427734, 79.25027465820312 43.54916763305664, 78.0655517578125 43.57444381713867, 77.50861358642578 43.58222198486328, 77.15860748291016 43.59416580200195, 76.96083068847656 43.59611129760742, 76.80860900878906 43.59749984741211, 76.32611083984375 43.599998474121094, 75.53028106689453 43.635833740234375, 74.19889068603516 43.68305587768555, 73.74833679199219 43.696109771728516, 71.25444793701172 43.54666519165039, 70.86027526855469 43.51527786254883, 68.36666870117188 43.29999923706055, 67.79360961914062 43.19333267211914, 67.44721984863281 43.12722396850586, 66.77833557128906 42.99638748168945, 66.09249877929688 42.85749816894531, 64.2249984741211 42.45333480834961, 64 42.40277862548828, 63 42.1683349609375, 62.78666687011719 42.11722183227539, 62.630001068115234 42.079444885253906, 62.079166412353516 41.94388961791992, 61.75 41.86166763305664, 60.654998779296875 41.578887939453125, 60.1974983215332 41.4716682434082)

while if I leave out the last 5 coordinates it will work:

LINESTRING (121.78333282470703 31.133333206176758, 121.7763900756836 31.140277862548828, 121.76249694824219 31.153888702392578, 121.7558364868164 31.16083335876465, 121.741943359375 31.1744441986084, 121.73500061035156 31.18138885498047, 121.71416473388672 31.20194435119629, 121.67972564697266 31.23611068725586, 121.63833618164062 31.2772216796875, 121.58999633789062 31.325000762939453, 121.45166778564453 31.462223052978516, 121.3758316040039 31.537500381469727, 121.28583526611328 31.626388549804688, 121.18916320800781 31.721389770507812, 121.08555603027344 31.822500228881836, 120.96138763427734 31.943889617919922, 120.81610870361328 32.08583450317383, 120.73332977294922 32.16666793823242, 120.6066665649414 32.176944732666016, 120.36360931396484 32.1966667175293, 120.09972381591797 32.21833419799805, 119.81444549560547 32.24166488647461, 119.76166534423828 32.24583435058594, 119.64305877685547 32.44722366333008, 119.46277618408203 32.75361251831055, 119.44833374023438 32.77777862548828, 117.21305847167969 36.41777801513672, 116.81639099121094 37.223331451416016, 116.37860870361328 38.11305618286133, 116.42500305175781 38.372501373291016, 116.57166290283203 39.19499969482422, 116.85444641113281 39.79944610595703, 116.76083374023438 40.02305603027344, 116.74305725097656 40.06611251831055, 115.95194244384766 41.95861053466797, 115.26277923583984 42.23611068725586, 105.59610748291016 41.20500183105469, 104.47333526611328 41.030555725097656, 95.19944763183594 41.86166763305664, 86.71611022949219 43.5433349609375, 84.3647232055664 43.34583282470703, 80.7330551147461 43.50027847290039, 79.8066635131836 43.53305435180664, 79.5594482421875 43.540279388427734, 79.25027465820312 43.54916763305664, 78.0655517578125 43.57444381713867, 77.50861358642578 43.58222198486328, 77.15860748291016 43.59416580200195, 76.96083068847656 43.59611129760742, 76.80860900878906 43.59749984741211, 76.32611083984375 43.599998474121094, 75.53028106689453 43.635833740234375, 74.19889068603516 43.68305587768555, 73.74833679199219 43.696109771728516, 71.25444793701172 43.54666519165039, 70.86027526855469 43.51527786254883, 68.36666870117188 43.29999923706055, 67.79360961914062 43.19333267211914, 67.44721984863281 43.12722396850586, 66.77833557128906 42.99638748168945, 66.09249877929688 42.85749816894531, 64.2249984741211 42.45333480834961, 64 42.40277862548828, 63 42.1683349609375, 62.78666687011719 42.11722183227539)

I checked that for many other lines with longer length it does work.
I would be inclined to think this is a bug and somehow if certain lines have a certain "spread" this creates an indexoutofbounds exception. I admit I 'm on an outdated version, but have noticed some other minor bugs on more recent versions which kept me from upgrading. Would expect it is quite likely that this is related to combination heatmaps+Linestrings in all versions.

Having a fix for this is crucial for me to continue to work with geomesa, so I hope one of the experts can do something with this ��
Thank you very much for the time!

Original JIRA Issue: https://geomesa.atlassian.net/browse/GEOMESA-3252

Key: GEOMESA-3252
Type: Bug
Priority: Major
Status: To Do
Resolution: Unresolved
Reporter: Joris Billen
Created: Thu, 8 Dec 2022 06:04:16 -0500
Updated: Mon, 18 Mar 2024 12:49:18 -0400
Affects Version(s): 3.0.0
Component(s): HBase

Comments:

5ac751bd48753e448a580d54 (Thu, 8 Dec 2022 11:26:46 -0500):

I learned that the error is only triggered when the map is completely zoomed out (which is the default for openlayers in geoserver when a map is loaded: it uses this envelope and fails= (scale 1: 279M)
envelope:-298.1531155978747,298.1531155978747,-163.12271684354704,163.12271684354704 If I then zoom in it will be ok, for instance for this envelope (scale 1:70M) BBOX=-26.03027337890626,-8.979492128906244,108.99902337890626,58.549804628906244.
As for my use, it is not necessary to be completely zoomed out, this could mean it works fine.
Will verify further with more data and in case it still holds, this ticket can be closed.

Reproduce the failure with the supplied linestring, heatmap SLD, and fully zoomed-out envelope. Start with org.locationtech.geomesa.process.analytic.DensityProcess.scala and the HeatmapSurface.addPoint stack entry, then compare the supplied Accumulo DensityProcessTest.scala case with the HBase path. Done means the zoomed-out heatmap handles the reported linestring without ArrayIndexOutOfBoundsException.