ckan / ckan/ckanext-validation

Schema editor

Open
#65 5 comments 0 reactions 0 assignees View on GitHub
v2
Dominant language
Python
Stars
30
Forks
33
PR merge metrics
No merged PRs in 30d

Description

## Goal

Allow publishers to define the schema of tabular data as part of the resource creation process, internally generating a Table Schema that gets stored as the `schema` field

## Prior work

@roll worked on an initial implementation a few years ago (ancient PR here: #25). It used [tableschema-ui](https://github.com/frictionlessdata/tableschema-ui) to render the UI, and under the hood [tableschema-js](https://github.com/frictionlessdata/tableschema-js) to infer the data schema and generate a Table Schema object

https://user-images.githubusercontent.com/200230/171604305-a2b55f0b-35ca-4065-8934-d116f0303b76.mp4

## Implementation options

UI-wise it is understood that we need update the component to use the new version,and that the UI/UX, form design, etc, needs to be definitely improved, but we have different options for the schema inferring part.

### Option 1: Keep the inferring in the client with tableschema-js

Pros:
* Better UX as the schema can be modified before uploading the file
* Easier to integrate in CKAN's resource creation flow, ie we use the component to generate a JSON Table Schema that directly gets submitted in the `schema` field
* File size doesn't seem to be a concern as I tested a 800Mb and the schema was inferred without issue, I assume it parses a subset of the rows

Cons:
* What are the plans for tableschema-js? Can we rely on it long term?
* How good is the inferring? I assume most if not all recent work on this area has gone to frictionless-py
* Would the schema generated by tableschema-js match the one generated by frictionless-py? Right now this is not important but I can imagine us having to implement some sort of server-side inferring for background jobs, etc, could we find inconsistencies between schemas generated by the two systems?

### Option 2: Use frictionless-py for the inferring

This of course requires the file to be uploaded to the server, as I don't think WASM-based solutions are ready for general production use.

Pros:
* We focus our efforts in just one Frictionless library (fricitonless-py), the one that is arguably better supported

Cons:
* 2-step process for creating a resource (3 if we count the previous dataset metadata step), file needs to be uploaded first, and then the schema can be returned to the user for tweaking.

#### Option 2a: Create the resource, infer the schema later

Users would create a resource normally and once is created we would infer the schema, redirect the user to a new step with the schema editor and allow them to tweak it further (but at this stage the inferred schema could already be stored in the created resource)

#### Option 2b: Upload the file first, infer the schema, create the resource later

This would be difficult to implement because right now uploads are closely tied to the actual resource, but we can imagine an implementation where the file is uploaded first (or linked), stored somewhere temporal, we run the inferring and return the result to the user, who then proceeds to create the resource, which is somehow linked to the uploaded file

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.