Search annotations in XDM

**Problem / Premise**

In the current systems in Adobe Search, there is no standardized way to indicate in events / otherwise as to what fields need to be indexed and made searchable, which ones need to be made available in the search index for querying etc. There is no easy way for clients to discover the nuances of the data sets the search APIs will expose in the responses to search queries.

**Need**
Given the above premise, there is a need to standardize the interfaces across layers with special reference to the data interfaces and specifications around XDM. The proposal covers the approach to standardize annotations in XDM to be able to easily define which fields for a particular object ( as represented by a XDM schema) are to be indexed. Another aspect is to be able to also move the definition of the search API response data aspects including which fields are filter-able, which ones fetch-able, which ones are sortable and which ones are searchable as well to XDM. Once implemented, the XDM spec will be the source of truth for determination of the characteristics of behavior to determine what data sets will be indexed. The XDM spec will also serve as the source of truth to determine the response characteristics for data sets as exposed through the search APIs.

**Approach**
We will extend the use of meta descriptors on XDM to define search annotations.
This is currently possible in two different ways : one to define them within the schema ( refer https://github.com/adobe/xdm/blob/master/docs/descriptors.md#embedding-schema-descriptors-in-a-schema ) , and the other being defining a standalone schema descriptor ( refer https://github.com/adobe/xdm/blob/master/docs/descriptors.md#example-of-defining-a-new-schema-descriptor )

The proposal is to use a combination of these two ways as enumerated above to define search annotations.

Search annotations proposed are listed below :

Field | Description and Usage | If Enum - enum literal values with description | Notes | Default value | Required ( Mandatory ) | Examples
-- | -- | -- | -- | -- | -- | --
search:indexed | Boolean indicating if the property is indexable / searchable | Not Applicable |   | false | true |  
search:store | Enum literals indicating if the raw value of the fields are stored in the index and can be fetched by defaultin response to search queries. Enum values : notStored, stored, storedAndFetchByDefault | notStored : Value of field is not stored in indexstored : Value of field is stored in indexstoredAndFetchByDefault : Value of field is stored in index and is fetched by default in search query responses. | This is an optimization to keep the search index small. | notStored | false |  
search:filterable | Boolean indicating if this property can be used as a filter in search queries | Not Applicable |   | false | false |  
search:sort | Enum literals indicating the type of sort applied to the indexed data.Enum values : notSortable, sorted, sortedCaseSensitive, sortedCaseInsensitive | notSortable : this field cannot be used for sortingsorted : used for non text data for which case sensitiveness does not mattersortedCaseSensitive : Indexed data is sorted in case sensitive mannersortedCaseInsensitive : Indexed data is sorted in case sensitive manner |   | notSortable | false |  
search:analyzers | Array of enum literals defining the analyzer to be used for the field ( Details below )Analysis is the process of converting text, like the body of any email, into tokens or terms which are added to the inverted index for searching. Search annotations provide for XDM schema to define basic analyzers to be used during indexing and running search queries.The enum literals provided for in the search descriptor for analyzer describe the type of analysis desired.List of enum literals to define analyzer typesEnum values : Refer Analyzer TypesNote 1 :More values can be added here in future for custom analysis where standard analysis does not work)Note 2 : Based on Search team's experience: Only 10-15% fields need more than 1 analyzer. Most of them belong to either standard analysis or no analysis. | Refer section below on analyzer types supported in XDM | These analyzers ( as defined in the enum literals ) are available out of the box and for use by all clients through XDM. More complex analyzers are possible to be used, but are not supported through XDM. For use of deeper analyzers, Search/Sensei will engage the clients through an on-boarding process with the Search team. This property on the descriptor allows for multiple enum literal values. This means that the same field may be subject to different types of analysis.For instance, a text field may be marked as both standard and noAnalysis. This means that the index will store two different manifestations of this field - one which is the raw value of the field ( noAnalysis ) and one which is the text derived after applying the standard analyzer. | noAnalysis | false | Standard text analyzer : For input text "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone.", this analyzer would break down the text into the terms [ the, quick, brown, foxes, jumped, over, the, lazy, dog, s, bone ] to be added to the inverted index for search. Standard Date analyzer : The date analyzer is a custom analyzer that supports formatting of dates in various ways to be stored in the index to support for querying and retrieval of analyzed date formats.Eg; Express UTC timestamp as full date as four digit year, two digit month of year, and two digit day of month (yyyy-MM-dd).Example for use of multiple analyzers on same field"tweet:first_name": { "type": "string", "title": "first_name", "meta:descriptors": [ { "@type": "xdm:searchdescriptor", "search:indexable": true, "search:filterable": true, "search:stored": "stored", "search:sort":"sortedCaseSensitive", "search:analyzer": ["noAnalysis", "standard"] } ] }In the above case is a real world example of first name property for a tweet described in XDM. Here the first_name field is being specified with a search meta descriptor with both analyzers no_analysis and standard defined. In this case, two versions of the first_name will be in the index and made available for search - one which is the raw first name ( corresponding to no_analysis ) and the other an analyzed version of the first name after applying the standard analyzer.For more examples, refer section below on analyzers.
search:defaultAnalyzer | Default analyzer to be used on the field |   | Since analyzer is an array, this field will indicate which one will be used to identify the field to be used in queries unless otherwise specified in the search API call.This will default to noAnalysis unless otherwise specified. | noAnalysis | false |  

Analyzer types

| Type | Description | Canonical name in Enum for search descriptor |
|-------------------|:----------------------------|:------------------------------------------------|
| Standard | This analyzer provides lower casing and standard grammar based tokenization. | standard |
| Generic language analyzer | Adds ( to the standard analyzer ) extended Unicode support using the ICU libraries, including better analysis of Asian languages, Unicode normalization, Unicode-aware case folding | langGeneric |
| Language analyzers | One for every supported analyzer | langFr, langTh, langDe, langJa, langZh |
| Prefix ( starts_with* analyzers ) | Prefix based analyzers that can determine if the text starts with a pattern and apply a set of character filters or tokenizers | startsWithDefaultAnalyzer , startsWithWhitespaceAnalyzer |
| Custom field analyzers | Analyzer that is specific to certain fields, and applies custom semantics specific to the field being analyzed ( for instance file name specific analyzers ) | filename, filenameExtension |

**Overrides**
For customer specific / scope specific annotations ( such as filterable, fetchable and sortable ) we can define them in configs on our end which comply to XDM schema and also define defaults in the respective schemas under each property itself.

Indexable could be only defined in the respective schemas itself ( default values for each annotation to be defined in relevant schemas of asset, entity etc against respective fields )

Wiki link with more details : https://wiki.corp.adobe.com/display/adobesearch/Search+annotations+in+XDM+-+Proposal

## What are the schemas that are affected by the issue
Asset, Search meta descriptors

## What are examples of products that are impacted by the issue
Search

Start with docs/descriptors.md sections on embedding schema descriptors and standalone schema descriptors, then inspect the affected Asset schemas and Search meta descriptors named in the issue. Compare the proposed search annotations and analyzer enums with existing XDM descriptor conventions and examples. Done means the affected schemas expose the agreed annotations and the XDM specification documents their defaults, overrides, and response behavior.

Written by the indexing model from the issue text.