Storing the subjects does not convert the vocabulary token to the value

Storing subjects with the data you get from @vocabularies/plone.app.vocabularies.Keywords does not work. The client get's a token and a label from @vocabularies and is supposed to send the token in the content POST/PATCH. For Choice schema fields that works as we do an automatic conversion from the vocabulary term's token to value. But the subject field is a schema.Tuple() with value_type=schema.TextLine(), and the vocabulary is not defined on the field, but as a annotation for the widget.

As a result a client sending the token will store the raw ascii encoded token instead of the value.

Related to #691

plone.app.dexterity.behaviors.metadata.ICategorization

@provider(IFormFieldProvider)
class ICategorization(model.Schema):

    # ...
    subjects = schema.Tuple(
        title=_(u'label_tags', default=u'Tags'),
        description=_(
            u'help_tags',
            default=u'Tags are commonly used for ad-hoc organization of ' +
                    u'content.'
        ),
        value_type=schema.TextLine(),
        required=False,
        missing_value=(),
    )
    directives.widget(
        'subjects',
        AjaxSelectFieldWidget,
        vocabulary='plone.app.vocabularies.Keywords'
    )

Vocabulary token -> value conversation currently happens only for Choice fields:

plone.restapi.deserializer.dxfields.ChoiceFieldDeserializer

@implementer(IFieldDeserializer)
@adapter(IChoice, IDexterityContent, IBrowserRequest)
class ChoiceFieldDeserializer(DefaultFieldDeserializer):
    def __call__(self, value):
        if isinstance(value, dict) and "token" in value:
            value = value["token"]
        if IVocabularyTokenized.providedBy(self.field.vocabulary):
            try:
                value = self.field.vocabulary.getTermByToken(value).value
            except LookupError:
                pass

        self.field.validate(value)
        return value

We need to find a way to lookup the value for the token when the vocabulary is assigned in an annotation. It should be done for other fields than IChoice too. Choice fields require the value to be in the vocabulary. Choice fields can't be used with SimpleVocabuary if the user should be able to create new entries on the fly like the subject field allows.

(Note: A workaround at the moment is to override theplone.app.vocabularies.Keywords vocabulary with a version that uses the value ass the token (without encoding/str() conversion). This seems to work in the legacy plone interface too: https://gist.github.com/csenger/9eaed84c20f332a03e77ff8b21b396fd)

Start with src/plone/restapi/deserializer/dxfields.py and ChoiceFieldDeserializer, then compare it with the ICategorization.subjects definition in plone.app.dexterity.behaviors.metadata.py. Determine how an annotation-supplied vocabulary can resolve tokens for non-Choice fields while retaining newly entered subject values. Done means token submissions store vocabulary values consistently beyond Choice fields.