astrofrog / astrofrog/numtraits

provide path for to_json/from_json of units for front-end

Open
#12 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
39
Forks
5
PR merge metrics
No merged PRs in 30d

Description

One of the marquee users of traitlets that users would encounter is widgets, either by way of `interact` or directly, as they provide one of the best ways to access the interactive Jupyter/IPython magic.

As such, for many science and engineering applications, having the units persist all the way to the front-end is enormously useful.

To talk to the front end, the entire meaning of the traitlet value at one time must be communicated over [serialization to JSON](https://github.com/ipython/ipywidgets/blob/4da7e4748b4c3dbb141ffbac364190b629671f66/ipywidgets/widgets/widget.py#L22): JSON is actually poorly represented vs other serialization formats, i.e. XML in providing rich types. The reference implementation, for example, of providing an identifier for `Widget` itself uses a magic string notation to an ephemeral id, `IPY_MODEL_`, which frontends can only manipulate very mechanically.

For units, serializing the number to a string would introduce a whole set of issues, as the pidgin language of every unit library is _slightly_ different.

A solution to this would be to utilize [JSON for Linking Data](http://json-ld.org/), and leverage extensive existing work into solving this non-trivial problem... or at least representing it in a way that is not overly-opinionated.

Consider:

``` python
from astropy import units as u

class Sphere(Widget):
radius = NumericalTrait(convertible_to=u.m)

s = sphere(radius=1.21)
print(s.radius)
>>> 1.21 meters
```

In JSON, and in JSON-LD, a number is a number:

``` json
{"radius": 1.21}
```

But in JSON-LD, a number can also be an object with an `@value`:

``` json
{"radius": {"@value": 1.21}}
```

This level of indirection gives us a place to put other metadata about the value.

The simplest possible approach would be to continue to treat the value as a literal, and only introduce `@type`:

``` json
{"radius": {"@value": 1.21, "@type": "meter"}}
```

Hooray, we've picked a type. But we've made up our own name for it. Where do we look the values up? What about derived units, domain, preferred display units, etc.?

Here's what it could look like by utilizing the [UN/CEFACT codes](http://www.unece.org/fileadmin/DAM/cefact/recommendations/rec20/rec20_Rev9e_2014.xls), which have been notionally adopted by [schema.org](http://schema.org/unitCode), a large driver of linked data adoption:

``` json
{
"radius": {
"@context": "http://schema.org/",
"@type": "QuantitativeValue",
"value": 1.21,
"unitCode": "MTR"
}
}
```

While we have said more explicitly (i.e. not unilaterally) that this value is of a type, and are playing by the rules of a standards body, this is somewhat unsatisfying:
- `MTR` is basically a random code from an excel spreadsheet
- derived units, if not already present in the several thousand already defined, would be difficult to describe some of the more interesting types possible, as a user may have created them on the fly through a series of manipulations.

To solve some of these issues, adoption of the [QUDT](http://www.qudt.org/) vocabularies would provide a more robust conceptual model:

``` json
{
"radius": {
"@context": [
"http://schema.org/",
{
"ex": "http://example.com#",
"radius": "ex:radius",
"unit": "http://qudt.org/1.1/vocab/unit#"
}
],
"@type": "QuantitativeValue",
"value": 1.21,
"unitCode": "unit:Meter"
}
}
```

This adds a "thing not a string" to the `unitCode`, itself which can be traced back to a robust set of models.
QUDT can also support vectors of exponentiated dimension types, etc.and comes with a very large library, written by and used within an organization with a seriously multi-scale perspective (NASA).
## Data Shapes

A whole other story. JSON-LD is pretty bad at labeling columns of arrays, and indeed URIs can't start with numerals. Some approach for listing columns and their types would be necessary.
## Implementation

TBD... probably something like ipywidgets, i.e. `numtraits.widget_serialization`, which exposed a `to_json` and `from_json` functions that consulted the canonical data format and called the appropriate things in the upstream unit library (i.e. astropy, pint).
## Dependencies

Generating and interpreting JSON-LD _requires_ no additional libraries. A JSON Schema library (which already ships with jupyter) would be sufficient to provide sufficient serialization robustness, even if it couldn't do full type-checking of the resources.

The canonical lists are available for download as XML or turtle, and these could be converted to canonical JSON.
## Front-end

Out of scope for this issue, but... in the near term, a set of base widgets (sliders, text boxes, etc.) which didn't simply fall over would be a good start.

As to serious implementations on the front-end parsing side of this, several quantity libraries exist including [math.js](http://mathjs.org/) and [quantities.js](https://github.com/gentooboontoo/js-quantities). There is nothing as flexible as any of the python implementations, but this could be an excellent driver for the creation of such a library, driven by a canonical representation format.
## Related:
- [This proposal](https://github.com/jupyter-incubator/proposals/pull/9) is mainly concerned with the DataFrame representation.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Implementation section and the referenced ipywidgets widget.py serialization code, then inspect the existing numtraits structure for a suitable widget_serialization entry point. Define the canonical JSON or JSON-LD representation and how to_json/from_json would consult upstream unit libraries such as astropy or pint; done requires an agreed format and a documented implementation scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter, python
Domain
backend-api-design, frontend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.