ckan / ckan/ideas

Nested Form Data Descriptions

Open
#191 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
39
Forks
1
PR merge metrics
No merged PRs in 30d

Description

CKAN's API includes lots of examples of nested data. CKAN's forms submit flat data using either `application/x-www-form-urlencoded` or `multipart/form-data` which is a little like a simple JSON object with only strings for keys and values (repeated keys may represent lists of 2+ strings).

This impedance mismatch leads to some complicated validation logic when we have multiple form elements representing nested data, especially when mapping validation errors to the original fields submitted. Some examples of this can be seen in https://github.com/ckan/ckanext-scheming, https://github.com/ckan/ckanext-fluent and https://github.com/espona/ckanext-composite

Here I'm presenting a way to describe the mapping between flat form data and nested JSON that would allow us to:
* build much more dynamic forms with JS
* extend certain forms by only overriding templates
* reduce some validation logic complexity
* enable our API to accept nested and file data at the same time.

## Nesting flat data with `_json_vars`

Take an example I'm working on right now: I need to build a form that accepts labels and notes fields for each column of a datastore table. The natural JSON representation for these would be a list of `{"label": ..., "notes": ...}` objects. To create that representation from my simple `text` and `textarea` inputs I include a hidden input field in the form:

```html

Label 1:


Notes 1:


Label 2:


Notes 2:

```

When the form is submitted, a new utility function will convert the posted data by replacing values in `_json_vars` with the fields of the same name, returning a single parameter "info" containing a list of two objects with "label" and "notes" populated.

This style extends to arbitrarily nested JSON objects and can be easily generated by JS on the front-end for dynamic forms, or by a small template helper for static forms.

## Mixing static JSON with `_json`

`_json_vars` covers cases where all values map to existing input fields. For cases where some of the data is already in JSON format we can use `_json`.

By adding the same utility function in the API controller, this command would create a dataset and upload two resource files as part of the same API call:
```bash
curl https://myckan --form '_json={
"name": "mydataset",
"resources: [{"name": "my data"}, {"name": "supporting docs"}]
}' --form '_json_vars = {
"resources": [{"upload": "res1"}, {"upload": "res2"}]
}' --form res1=@data.csv --form res2=@docs.doc
```

The values in `_json` will be updated by the values from `_json_vars` to insert files into the resources at the same time they are being created.

`ckanapi.RemoteCKAN` can be updated to generate `_json` and `_json_vars` automatically when nested file objects are passed, making client code simpler (currently `RemoteCKAN` raises an error because this is only possible with `LocalCKAN`).

## Preventing conflicts with `_json_hash`

Our forms create another problem when being used concurrently by many clients: Current values are loaded into the form then some time later accepted blindly when submitted. This can easily cause values to be accidentally reverted when two people are using the same form, or one person's changes to be overwritten when another user makes a change. Ideally we would avoid updating fields that we haven't changed and send an error to a user when one of their changes conflicts with another users' changes.

`_json_hash` is a value that will be generated by the controller/view and passed to the template to be rendered as a hidden field to be passed back on submit (JS could be used to generate these hash values for dynamic forms). The controller/view will compare `_json_hash` values with the fields submitted to see which fields have been modified by the client. For fields that have been modified by the client it may then use the hashes to ensure that the values being updated in the database match the ones that appeared in the form, and if not a validation error can be raised.

This feature can be phased in slowly, implemented in just the most important forms first. The controller/view can decide on the granularity and which values are excluded. Documenting a specific format is only required for dynamic forms, but using a JSON representation makes is easier for dynamic JS to generate the necessary hash values.

If I decide to hash the labels and notes together on my datastore form I might use a format like:
```python
_json_hash = {"info":[
"45b6785d13dd73a9",
"dc5e0c11acb38306"
]}
```

With the right controller/view logic this would allow two clients to update different fields separately at the same time but if they both change the same field the second one to submit would get an error.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the form and API controller handling described for `_json_vars`, `_json`, and `_json_hash`, then inspect `ckanapi.RemoteCKAN` for its nested-file behavior. Define the implementation scope and tests for nested form conversion, mixed JSON and file uploads, and conflict detection before coding; done means the supported submission patterns work without breaking existing forms.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, python
Domain
api, backend, frontend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.