assemblee-virtuelle / assemblee-virtuelle/semapps

Use a single dataset per data provider / frontend app

Open
#1,393 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
103
Forks
14
Avg merge
1m
Merged PRs (30d)
2

Description

**Issue**
I'm about to refactor the data provider to use ldo in general.

This comes with improvements such as
- typed objects
- link-traversal (auto-dereferencing required resources)
- the possibility to keep track of changes per object using transactions
- translating changes to diffs or SPARQL queries
- stores all data in RDF.js datasets, which store data as quads.

What ldo doesn't implement (yet) but which we can augment:
- Schema Validation
- Caching
- Pagination

**My question**
Can we store all data in one RDF dataset or should we create one dataset per resource?

In practice a resource URI translates to the graph URI. So storing everything in one RDF dataset should be fine in the vast majority of cases.

**Approach to using a single dataset**
The dataprovider needs to be refactored so that it queries resources accordingly, if they are not cached already or stale. This requires a slightly different caching approach.
Also, caching and filtering will need to be addressed.

In either case, we need to add/refactor pagination and schema validation.

**Possible Issues**
Where do you see issues with using one dataset for all frontend data?

- Also, we might run into issues when the graph URI does not translate to the resource URI: JsonLd objects can store information about graphs that do not belong to the same resource. E.g. I fetch https://foo.bar which returns a jsonld resource which contains a graph with id https://graph.id - should data from that graph as defined here be stored in the general dataset? What happends if I fetch https://graph.id and the data is contradictory?
- Refactoring filtering / queries could be challenging. We might have to come up with a strategy to decide on whether we need to fetch ldp resources or use a sparql query (if sparql is available).

**Benefits**
Currently, there is a lot of redundancy in the data fetched so it would probably bring performance gains and reduce server load.

I think that the single-dataset approach brings us closer to a unified framework which supports both the solid and the NextGraph sparql-based world.

Also, I suppose that it reduces complexity because you don't have to think about the dataset where data is stored in (almost) every case.

**What next?**
I would like to try using one dataset only with the `use(typed)Collection` implementation first and see how it goes.

---

@srosset81 do you have an opinion here?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the data provider and the use(typed)Collection implementation mentioned in the issue. Review how RDF.js datasets, resource queries, caching, filtering, pagination, and schema validation are currently handled. Done means deciding and validating whether one dataset can support the frontend data provider without contradictory graph data or broken resource and query behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
data, frontend
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.