locationtech / locationtech/geomesa
Spark - 'count' does not always return the correct result
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 1.5k
- Forks
- 446
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 31
Description
Spark - 'count' does not always return the correct result
This may only affect the FSDS. There are many different ways to run a count - some/most of them return 0 even when running a regular query with the same filter returns results. I've generally had success with .rdd.count on a dataframe.
Original JIRA Issue: https://geomesa.atlassian.net/browse/GEOMESA-2597
Key: GEOMESA-2597
Type: Bug
Priority: Major
Status: To Do
Resolution: Unresolved
Reporter: Emilio Lahr-Vivaz
Created: Sun, 31 Mar 2019 17:11:40 -0400
Updated: Wed, 17 Jan 2024 12:00:36 -0500
Affects Version(s): 2.1.0, 2.2.0, 2.1.1, 2.3.0, 2.1.2, 2.2.1, 2.2.2, 2.1.3
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No implementation files or tests are named. Start by reproducing the issue in the FSDS with a DataFrame count and the same filter used by a regular query, then compare it with .rdd.count; done means count consistently matches the query results for the affected versions and execution paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100