Investigate thread pools for parallel table scans
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Quiet
- Tech stack
- scala
- Domain
- databases, distributed-systems, performance
Research direction
Start by reading HBaseQueryPlan and AccumuloQueryPlan to compare their sequential and parallel table-scan paths. Inspect how the HBase batch scanner creates and manages query threads, and how Accumulo's BatchScanner manages them. Done means establishing whether a shared pool can safely support parallel scans and defining the resulting implementation scope.
Written by the indexing model from the issue text.
Description
Investigate thread pools for parallel table scans
In HBaseQueryPlan and AccumuloQueryPlan, we have two modes for scanning multiple tables (used by partitioned data stores): sequential and parallel. Currently the downside of parallel scans is that each scan will use the full number of query threads. It might be better to always scan in parallel if we can keep a single thread pool for all the scans. In Accumulo, this may be tricky as threads are managed by the BatchScanner. In HBase, we have our own batch scanner so we could modify that to use a shared thread pool.
tagging for comment: Andrew HulbertJames HughesAnthony Fox
Original JIRA Issue: https://geomesa.atlassian.net/browse/GEOMESA-2436
Key: GEOMESA-2436
Type: Improvement
Priority: Major
Status: To Do
Resolution: Unresolved
Reporter: Emilio Lahr-Vivaz
Created: Fri, 12 Oct 2018 10:23:52 -0400
Updated: Fri, 12 Oct 2018 10:23:52 -0400
- Dominant language
- Scala
- Stars
- 1.5k
- Forks
- 446
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 31
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from locationtech/geomesa
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
locationtech/geomesa#10761 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
locationtech/geomesa#10324 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
locationtech/geomesa#10111 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
locationtech/geomesa#10052 ·
-
Difficulty 1/5 1-3 hours Newbie friendliness 78/100
locationtech/geomesa#9998 ·
All issues in locationtech/geomesa
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
-
requires-triage
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
apache/datafusion-comet#6060 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
lichess-org/lila#21756 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
salesforce/evalon#10 ·