azavea / azavea/osmesa

Create deployment OSMesa with Ingest and Zeppelin notebook capabilities

Open
#3 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
81
Forks
26
PR merge metrics
No merged PRs in 30d

Description

Develop the necessary terraform scripts to bring up an OSMesa stack for the Ingest and Global Analytics - Zeppelin Notebook components.

- [ ] Bring up an ephemeral, spot-instance based cluster that has HBase on S3 enabled
- [ ] Run all necessary GeoMesa configuration against the cluster (only on brand new clusters; other instances of ephemeral clusters pointing at an existing S3 backend do not need to do this)
- [ ] Run an ORC ingest (could be Makefile driven with `aws emr add-steps`)
- [ ] Tear down the ingest cluster
- [ ] Bring up an Analytic cluster with HBase on S3, pointed at a pre-existing cluster, _in read only mode_, with the Zeppelin application enabled
- [ ] Place all necessary JARs onto the master node, running a Zeppelin notebook, and doing simple tests against the data ingested.

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are named. Start by locating the repository's Terraform configuration and the Makefile or `aws emr add-steps` workflow, then map the ingest and analytic cluster steps to the checklist. Done means the ephemeral ingest flow, read-only Zeppelin analytic cluster, required JAR placement, and simple notebook data tests all work.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, scala, spark, terraform
Domain
cloud, data-engineering, distributed-systems, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.