Integrate Halyard with SANSA Stack
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 114
- Forks
- 18
- PR merge metrics
- No merged PRs in 30d
Description
Halyard is powerful distributed triplestore, instantly answering majority of SPARQL queries, however weak in some complex operations (like ORDER BY and GROUP BY) and complicated to implement a custom code that goes beyond SPARQL.
SANSA Stack (and similar Spark-based SPARQL frameworks) seem to be complimentary to Halyard - powerful in ordering, aggregations, and easy to integrate custom transformation logic into the pipe, however slow in ad-hoc SPARQL queries, and unable to form SPARQL Endpoint.
The idea is to provide a hybrid solution, where SANSA Stack (or any other Spark framework) can directly use Halyard data and Halyard query engine as a (distributed) source of RDF data for further processing.
- Minimal implementation is to provide Halyard library for Spark, so SANSA Stack can directly consume the Halyard data (read the RDF data directly from HBase) and can call Halyard SPARQL Query Engine (consume results from Halyard SPARQL Graph Query locally and directly).
- Integrated solution would require to include Halyard as a Service Provider in SANSA SPARQL Query Engine, so hybrid access to Halyard data from Sansa would be available inside SANSA SPARQL as a Federated Service Provider.
- Optimal solution would also include transparent integration of Halyard SPARQL parallelization (similar to
halyard:forkAndFilterByfunction used in Halyard BulkExport), so Spark engine would be able to directly manage Halyard parallelization (transparently for user).
This is an idea of potential synergy effect of Halyard and SANSA Stack, that seems to be worth to test.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing Halyard's HBase RDF access and SPARQL query engine, then examine the SANSA Stack Spark integration points described in the issue. Determine whether the minimal library, federated service provider, or transparent parallelization is in scope; done would require an agreed implementation scope and an integration that supports the selected hybrid access path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- backend, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100