Support semantic relationship between PublishedData and Dataset

Summary

SciCat supports publishing data, by allowing a user to collect one or more datasets, providing metadata (mostly bibliographic) and mint a DOI. This published data process creates a PublishedData record. This record contains the links to the set of underlying dataset. Currently, this link is simply a list of identifiers of the member datasets: the pidArray property.

Although this behaviour is functional, there is some information that is currently not captured and that (I think) would be useful.

Specifically, the current publishing process allows the user to indicate a set of datasets to be published, but it doesn't allow the user to specify the context of those datasets within the published data. As a guide, this context would answer the question: why is this dataset included in this published data?

This context might be described using a complex and powerful description language; however, the simplest option would be a simple, short text field.

Here are a couple of examples, along with possible context/labels:

• a published dataset might include datasets with different roles:
  • a calibration dataset (labelled as calibration),
  • a measurement dataset (labelled as measurement),
  • a zero/dark-field dataset (labelled as darkfield),
  • a reference dataset (labelled as reference),
  • ...etc...
• a published data might include datasets with different samples:
  • a dataset from the first sample (labelled as sample #1),
  • a dataset from the second (related) sample (labelled as sample #2),
  • ...etc...

This problem is, perhaps, most keenly seen on the DOI landing page. That landing page lists the component datasets as a series of hot links that the user can select (in their browser). Doing this directs their browser to a page that describes this dataset-specific. However, on the DOI landing page, the member datasets are identified only by their SciCat-internal identifier. This makes it very hard for the user to understand "why" these datasets are being published.

Steps to Reproduce

1. enter multiple datasets
2. create a publication data, from these datasets.
3. point the browser at the DOI landing page
4. from information on the DOI landing page, try to figure out which dataset is which.

Current Behaviour

DOI landing page shows only the internal identifier of the dataset.

Expected Behaviour

Landing page shows a helpful (short) summary of the member datasets.

Details

Datasets already have some descriptive metadata for human consumption: description datasetName and comment. Of these, perhaps datasetName would be the most obvious choice for describing the dataset; however, this might be rather longer than desired.

However, there is a deeper issue is that the context is something not from the dataset itself but rather the motivation behind the dataset's selection within the PublishedData (why this dataset?)

For example, a dataset might be included in several PublishedData records. In one PublishedData, the dataset might be identified simply as reduced dataset (with the PublishedData including other datasets, identified as measurements, calibration data, etc). In another PublishedData, the same dataset might be identified as reference data (used in a comparison with some other dataset). In yet another PublishedData, the same dataset might be identified as perovskite sample (used in a comparison with datasets from other samples).

In more concrete terms, I'm proposing that the PublicData records are updated, so the pidArray property is replaced with a property containing a list of objects.

Here is an example pidArray value

[
    "1012375812",
    "12375747124",
    "723465883234"
]

Here is the same example, but showing a pidArray-replacement property:

[
    {
         "pid": "1012375812",
         "context": "measured data"
    },
    {
         "pid": "12375747124",
         "context": "reference data"
    },
    {
         "pid": "723465883234",
         "context": "calibration data"
    }
]

An automated migration process cannot fill in the context fields automatically. One approach would be to make the context field optional; where this property would be missing after the migration. The landing page would use the current behaviour (showing the dataset identifier) if the context property is missing.

Start by tracing how PublishedData records store pidArray and how the DOI landing page presents member datasets. Define the replacement object shape, migration behavior for missing context values, and compatibility expectations before identifying the affected models, APIs, and tests; done means contextual member descriptions appear without breaking existing records.