Merck / Merck/Halyard

Integrate Halyard with SANSA Stack

Open
#71 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement opportunity
Dominant language
Java
Stars
114
Forks
18
PR merge metrics
No merged PRs in 30d

Description

Halyard is powerful distributed triplestore, instantly answering majority of SPARQL queries, however weak in some complex operations (like ORDER BY and GROUP BY) and complicated to implement a custom code that goes beyond SPARQL.
SANSA Stack (and similar Spark-based SPARQL frameworks) seem to be complimentary to Halyard - powerful in ordering, aggregations, and easy to integrate custom transformation logic into the pipe, however slow in ad-hoc SPARQL queries, and unable to form SPARQL Endpoint.

The idea is to provide a hybrid solution, where SANSA Stack (or any other Spark framework) can directly use Halyard data and Halyard query engine as a (distributed) source of RDF data for further processing.

  1. Minimal implementation is to provide Halyard library for Spark, so SANSA Stack can directly consume the Halyard data (read the RDF data directly from HBase) and can call Halyard SPARQL Query Engine (consume results from Halyard SPARQL Graph Query locally and directly).
  2. Integrated solution would require to include Halyard as a Service Provider in SANSA SPARQL Query Engine, so hybrid access to Halyard data from Sansa would be available inside SANSA SPARQL as a Federated Service Provider.
  3. Optimal solution would also include transparent integration of Halyard SPARQL parallelization (similar to halyard:forkAndFilterBy function used in Halyard BulkExport), so Spark engine would be able to directly manage Halyard parallelization (transparently for user).

This is an idea of potential synergy effect of Halyard and SANSA Stack, that seems to be worth to test.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing Halyard's HBase RDF access and SPARQL query engine, then examine the SANSA Stack Spark integration points described in the issue. Determine whether the minimal library, federated service provider, or transparent parallelization is in scope; done would require an agreed implementation scope and an integration that supports the selected hybrid access path.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
backend, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.