dice-group / dice-group/Squirrel

Improve parsing robustness

Open
#64 3 comments 0 reactions 0 assignees View on GitHub
component:worker priority:high type:enhancement
Dominant language
Java
Stars
25
Forks
19
PR merge metrics
No merged PRs in 30d

Description

## Description

The [LOD Laundromat](http://lodlaundromat.org/) has shown that
* Online several RDF datasets contain errors
* They still can be parsed with robust parsers

The goal of this issue is to adapt the robustness of the LOD Laundromat to Squirrel.

## Solution
* The LOD Laundromat has been programmed in Prolog so we can not use the code "directly" in our Java crawler
* It might be possible to have a step between the fetcher and the analyzer which makes sure that the RDF is valid, i.e., instead of `Fetcher -> File A -> Analyzer` we could have `Fetcher -> File A -> RDF Checker -> File B -> Analyzer` where the RDF Checker could be a Prolog program using the LOD Laundromat code.

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no files or tests; begin by tracing the existing Fetcher -> File A -> Analyzer flow in the Java crawler. Compare it with the proposed RDF Checker step and determine how malformed RDF should be handled before the Analyzer; the issue does not define a concrete completion test or implementation boundary.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.