stackabletech / stackabletech/demos
HBase Spark Demo
Open
@snocke is already working on this.
Since Aug 30, 2022.
- Dominant language
- Jupyter Notebook
- Stars
- 3
- Forks
- 13
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 2
Description
Loading data into HBase is not trivial. We want the demo to show how this can be done and to provide some guidance and best practice.
Aims
- Load data row by row (NiFi)
- Batch processing CSV files (MapReduce)
- Direct load of HFiles
- Test HBase Spark connector (https://github.com/stackabletech/stackablectl/issues/71)
Tasks
- Load data into HDFS from S3
- Parse CSV and create HFiles
- Load incremental HFiles into HBase
- Load a streaming data source into HBase
- Stackable cluster configuration
- Verify the data is there (sanity check) using HBase shell
- Create Phoenix view over table
- Configure Phoenix as a data source in SuperSet
- Create a visualisation using Phoenix JDBC and SuperSet
- Query HBase using Spark HBase connector
## Learning Points and Challenges
- Where does DistCP and HBase bulk load run, given there is no YARN cluster?
- Are these jobs scalable?
- Can we go near real time dashboards in Grafana and see instant updates
- Stress testing
- Test HBase region management - can we watch this in real time as part of a demo?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.