ckan / ckan/ckanext-scheming

Validators ignored for fields with repeating subfields

Open
#394 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
102
Forks
182
PR merge metrics
No merged PRs in 30d

Description

## Environment

**CKAN version:** 2.10.1
**ckanext-scheming version:** release-3.0.0

## Description

Our CKAN implementation is being used to index datasets across a variety of domains. We have a schema that defines a core set of fields that all datasets need to specify. To support flexibility for dataset authors to add additional key-value metadata that is outside of the required core set of fields, our schema also has a field called `custom_fields` that uses `repeating_subfields` to allow users to specify additional key-value pairs to associate with their dataset. The `key` subfield has a validator that makes sure that the key's name meets certain requirements:

```yaml
- field_name: custom_fields
label: Custom Fields
repeating_label: Custom Field
repeating_subfields:
- field_name: key
label: Key
validators: key_validator
required: true
- field_name: value
label: Value
required: true
```

We want to block users from specifying the same `key` value in multiple entries in `custom_fields`. The obvious way to do this is to define a validator function, `keys_are_unique()`, that can be applied to `custom_fields` to verify that the keys are unique.

```yaml
- field_name: custom_fields
label: Custom Fields
validators: keys_are_unique
repeating_label: Custom Field
repeating_subfields:
- field_name: key
label: Key
validators: key_validator
required: true
- field_name: value
label: Value
required: true
```

After trying this & spending some time in my debugger, I discovered that the extension initially saves the top-level field's validator, but then replaces it with the subfields' validator(s). When this happens, the data type of the validators list also changes from `list[str]` to `dict`. I haven't had time yet to dig into why/how this works in the grand scheme of CKAN dataset creation, but that's probably my next step.

One example of this behavior is in the `_field_validators()` function in `ckanext/scheming/plugins.py`:

https://github.com/ckan/ckanext-scheming/blob/8646a9dce79aa0b5a46274271ca6330dc0870b92/ckanext/scheming/plugins.py#L582-L608

As a user, this behavior is unexpected -- I would expect that validators can be applied to *both* the top-level & sub-level fields. I think that it's worth updating the documentation for `repeating_subfields` and/or `validators` to explain that if a field has `repeating_subfields`, then `validators` for that field are ignored in favor of validators defined for the subfields.

I also have two specific questions, any insight would be appreciated:

1) We are trying to allow users to add an arbitrary number of "extra" key-value pairs to a dataset that is defined by a `ckanext-scheming` schema. Is our current approach (`custom_fields` field with `repeating_subfields`) the "correct" way to do what we are trying to do, or is there another method that is better supported, either by `ckanext-scheming` or by CKAN itself?

2) Assuming that what we are doing is correct, roughly how difficult would it be to modify `ckanext-scheming` to support validation of *both* the subfields and the top-level field? Any tips on what we would need to do to support that? It almost seems like this would require changes to the way the data is modelled, which I assume would be nontrivial.

Contributor guide

No contributing guide indexed for this repository

Research direction

Read _field_validators() in ckanext/scheming/plugins.py around lines 582-608 and trace how validators are stored for fields with repeating_subfields. Determine whether both top-level and subfield validators can be preserved without changing the data model; done means the behavior is either supported with coverage or clearly documented as intentional.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.