Correctify Identifier definition to conform to Unicode standard in "Lexical structure"

Background
--

When the C\# language specification was written (that which eventually became [ECMA-334, 1st Edition](http://www.ecma-international.org/publications/files/ECMA-ST-ARCH/ECMA-334%201st%20edition%20December%202001.pdf), it incorporated the rules for "Identifiers" as defined in "The Unicode Standard, Version 3.0" (published January, 2000). That definition (for the most part), is as follows (taken from the C\# language specification; I added the comments translating to the General Categories):

```antlr
identifier_or_keyword
: identifier_start_character identifier_part_character*
;

identifier_start_character
: letter_character /* General Categories: Lu, Ll, Lt, Lm, Lo, or Nl */
| '_'
;

identifier_part_character
: letter_character /* General Categories: Lu, Ll, Lt, Lm, Lo, or Nl */
| decimal_digit_character /* General Category: Nd */
| connecting_character /* General Category: Pc */
| combining_character /* General Categories: Mn or Mc */
| formatting_character /* General Category: Cf */
;
```

Since then, the compiler has been updated to use definitions from newer versions of Unicode. This has resulted in not only more characters being available for identifiers, but in some cases has also resulted in some breaking changes, as noted below (taken from [here](https://github.com/dotnet/roslyn/blob/fab7134296816fc80019c60b0f5bef7400cf23ea/docs/compilers/CSharp/Unicode%20Version.md) ):

> Unicode Version Change in C\# 6
> --
>
> The Roslyn compilers depend on the underlying platform for their Unicode behavior. As a practical matter, that means that the new compiler will reflect changes in the Unicode standard.
>
> For example, the Unicode Katakana Middle Dot "・" (U+30FB) no longer works in identifiers in C# 6. Its Unicode class was Pc (Punctuation, Connector) in Unicode 5.1 or older, but it changed to Po (Punctuation, Other) in Unicode 6.0.

^{Please see **Notes** section regarding code point U+30FB}

Issues
--

1. The "Lexical structure" document still states the original version of Unicode that was used back in 2000, even though the compiler has been updated to use a newer version of Unicode. The following is found immediately below the ANTLR definition of Identifiers:
> For information on the Unicode character classes mentioned above, see The Unicode Standard, Version 3.0, section 4.5.

1. The "Identifier" definition was never the official definition, which in Unicode 3.0 (the version that the spec states is being used, or at least _was_ used at that time) was:

```antlr
::= ( | )*

::=
::=
```

The net result is the same (well, except that the C\# spec added the customization of the "Low Line" character -- i.e. underscore -- as a valid starting character). However, including `identifier_part` is an unnecessary introduction of non-standard terminology (as far as I can tell, it was only ever found in the "PropList.txt" file). Unless there is a technical reason to deviate, the C\# specification should use the official definition(s) as stated in [Unicode Technical Report dotnet/csharplang#31: IDENTIFIER AND PATTERN SYNTAX](http://www.unicode.org/reports/tr31/) (though it's _not_ the syntax shown directly above, more on this in a moment).

1. While characters do sometimes get re-classified into different General Categories, and definitions of derived properties sometimes change, neither of those events should ever lead to a breaking change with regards to identifiers. Unicode has a stated policy of preventing such things. The following quote is from the "[Unicode Character Encoding Stability Policies](http://www.unicode.org/policies/stability_policy.html#Identifier)" document

> ### Identifier Stability
>
> **Applicable Version: Unicode 3.0+**
>
> **All strings that are valid default Unicode identifiers will continue to be valid default Unicode identifiers in all subsequent versions of Unicode. Furthermore, default identifiers never contain characters with the Pattern\_Syntax or Pattern\_White\_Space properties.**
>
> If a string qualifies as an identifier under one version of Unicode, it will qualify as an identifier under all future versions. The reverse is not true—an identifier under Version 5.0 may not be an identifier under Version 4.0—it may contain a character that was unassigned under Unicode 4.0, or (very rarely) a Unicode 4.0 character that was not an identifier character in Unicode 4.0, but became one in Unicode 5.0

They even included a section on stability starting in [revision 5 of TR \#31](https://www.unicode.org/reports/tr31/tr31-5.html#AutoNumber6), back in 2005. This section shows that once a character is permitted in identifiers, it will always be permitted in future versions of the Unicode Standard.

So, what happened? On the surface it might appear to be a simple case of the definition(s) changing. For example:
1. in [Unicode 4.0](https://www.unicode.org/reports/tr31/tr31-1.html#Default_Identifier_Syntax) (in 2003) the definition of `` was updated to include `Other_ID_Start = true`.
1. In [Unicode 4.1](https://www.unicode.org/reports/tr31/tr31-3.html#Default_Identifier_Syntax) (in 2004) the term `` was changed to ``, and the definition of `` was updated to include `Other_ID_Continue = true`.
1. In [Unicode 4.1](https://www.unicode.org/reports/tr31/tr31-5.html#Default_Identifier_Syntax) (in 2005) added derived properties: `XID_Start` and `XID_Continue`.
1. In [Unicode 5.0](https://www.unicode.org/reports/tr31/tr31-6.html#Default_Identifier_Syntax) (in 2006) elevated `XID_Start` and `XID_Continue` as "preferred" over `ID_Start` and `ID_Continue`.
1. In [Unicode 5.1](https://www.unicode.org/reports/tr31/tr31-9.html#Default_Identifier_Syntax) (in 2008) changed the definitions slightly to include `--[:Pattern_Syntax:]--[:Pattern_White_Space:]`.

But the real issue is that, due to such changes across versions of the Unicode Standard, attempting to derive the correct list of characters based on the rules and Categories is quite error prone, and even more so now that the rules are no longer based on just General Categories. Well, Unicode 3.0 does not seem to define how to handle determining which characters have a particular derived property, so it makes sense that the given formula would be used. However, starting no later than in version 5.1 (in 2008), the Unicode Standard, in [Unicode Standard Annex dotnet/csharplang#44: UNICODE CHARACTER DATABASE](https://www.unicode.org/reports/tr44/tr44-2.html#Simple_Derived_html), states explicitly that implementations should a) use the provided list of characters, and b) not derive the list based on the provided algorithm. The link for UAX # 44 in the previous sentence goes to revision 2 (for Unicode 5.1 in 2008), but the following quote is taken from revision 8 (for Unicode 6.1 in 2012) since it was re-worded to be clearer and was the first to include an example (a very pertinent one, in fact), and has not changed since, at least not through revision 24 (emphasis added):

> _Implementations should simply use the derived properties, and should not try to rederive them from lists of simple properties and collections of rules, because of the chances for error and divergence when doing so._
>
> _Definitions of property derivations are provided for information only_, typically in comment fields in the data files. Such definitions may be refactored, refined, or corrected over time. These definitions are presented in a modified set notation, expressed as set additions and/or subtractions of various other property values. For example:
> ```
> # Derived Property: ID_Start
> # Characters that can start an identifier.
> # Generated from:
> # Lu + Ll + Lt + Lm + Lo + Nl
> # + Other_ID_Start
> # - Pattern_Syntax
> # - Pattern_White_Space
> ```
> When interpreting definitions of derived properties of this sort, keep in mind that set subtraction is not a commutative operation. Thus "Lo + Lm - Pattern_Syntax" defines a different set than "Lo - Pattern_Syntax + Lm". The order of property set operations stated in the definitions affects the composition of the derived set.
>
> If there are any cases of mismatches between the definition of a derived property as listed in DerivedCoreProperties.txt or similar data files in the UCD, and the definition of a derived property as a set definition rule, the explicit listing in the data file should always be taken as the normative definition of the property. As described in Stability of Releases the property listing in the data files for any given version of the standard will never change for that version.

Remedies
--

The following needs to happen in order to conform to the Unicode Standard:

1. State the actual version of the Unicode Standard that the rules and definitions are being taken from. Stating something like "a newer version" is not acceptable.

> [UAX31-C1](https://www.unicode.org/reports/tr31/#C1). _An implementation claiming conformance to this specification shall identify the version of this specification._

1. State which requirements the compiler will follow:

> [UAX31-C2](https://www.unicode.org/reports/tr31/#C2). _An implementation claiming conformance to this specification shall describe which of the following requirements it observes:_ ...

There is a list of requirements. The C\# specification probably won't meet all of them, but it doesn't need to.

1. Change the definition of "Identifiers" to use `XID_Start` and `XID_Continue`.

> The XID_Start and XID_Continue properties are improved lexical classes that incorporate the changes described in _Section 5.1, [NFKC Modifications_](https://www.unicode.org/reports/tr31/#NFKC_Modifications) . They are recommended for most purposes, especially for security, over the original ID_Start and ID_Continue properties.

There aren't many differences between the "`ID_`" and "`XID_`" versions. For Unicode 12.1, `XID_Start` has 23 fewer characters than `ID_Start`, and `XID_Continue` has only 19 fewer characters than `ID_Continue`.

1. Define a "profile" to properly describe extensions (i.e. customizations) made to the "Default Identifier Syntax". Such customizations would include:
1. Adding "Low Line" (U+005F; i.e. underscore) to `identifier_start`
1. Potentially adding any characters back in from `ID_Start` that were removed in `XID_Start` that might cause this to be a breaking change.
1. Potentially adding any characters back in from `ID_Continue` that were removed in `XID_Continue` that might cause this to be a breaking change.

1. New definition of Identifiers (at bare minimum; not including any characters being added back from non-`X` definitions for backwards compatibility) should be something like:

```antlr
identifier_or_keyword
: identifier_start_character identifier_continue_character*
;

identifier_start_character
:
| '_'
;

identifier_continue_character
:
;
```

1. Incorporate the `XID_Start` and `XID_Continue` _lists_ from the "DerivedCoreProperties.txt" file, available on Unicode.org via public FTP. Here is the link to the file for the "latest" version of the Unicode Standard, but it is possible that an earlier version is used. If that is the case, then please be sure to use the intended version of the file (which is stated at the top of the file):

ftp://www.unicode.org/Public/UCD/latest/ucd/DerivedCoreProperties.txt

Please do _not_ attempt to use the rules / formula to derive the list of characters.

Notes
--

1. Regarding code point [U+30FB (Katakana Middle Dot)](https://unicode.org/cldr/utility/character.jsp?a=30fb) as noted in the "**Unicode Version Change in C\# 6**" quote above (final item in the **Background** section): this particular code point appears to be an anomaly and so probably shouldn't be used as evidence of breaking changes. Here is what I discovered about U+30FB:
* It was [discussed by a working group on 2000-03-02](http://www.unicode.org/mail-arch/unicode-ml/Archives-Old/UML021/0819.html):

> U+30FB KATAKANA MIDDLE DOT is a different animal entirely. It is connecting punctuation, used to bind together two-part (or multiple-part) katakana representations of foreign words or names. In a programming context, it can be seen as functioning something like the use of "\_" in C to form a multi_word_identifier, for example. The Japanese explicitly requested that it be allowed in identifiers.
* It was originally categorized as "Pc", at least in versions 3.0 through 4.0.1
* It qualified as being in the "identifier_extend" list due to being "Pc", at least in versions 3.0 and 3.0.1
* It was included in the `ID_Continue` list (in the **DerivedCoreProperties.txt** file, introduced in version 3.1), at least in versions 3.1 through 4.0.1
* Its general category changed from "Pc" to "Po" in Unicode version 4.1 (not version 6.0)
* It was no longer included in the `ID_Continue` list starting in version 4.1
* For some reason (which I have not yet discovered) it was _not_ included in the `Other_ID_Continue` ["Contributory" property](http://www.unicode.org/reports/tr44/#Contributory_Properties) (introduced in version 4.1; in the **PropList.txt** file) which is designed to support backwards compatibility and contains "grandfathered" characters that had once been in the `ID_Continue` list but are no longer in the current version.
* Recategorization alone cannot be the reason that it's no longer valid due to a) it _should_ be in the `Other_ID_Continue` list, and b) being "Po" isn't a guarantee of exclusion due to "U+00B7 MIDDLE DOT" being in `XID_Continue` (starting in version 3.1, as well as it and "U+0387 GREEK ANO TELEIA" being in both `ID_Continue` and `XID_Continue` starting in Unicode version 5.1, via `Other_ID_Continue`).
* It can easily be added via customization, and is even in the "Candidate Characters for Inclusion in Identifiers" list (Table 3) in section "[2.4 Specific Character Adjustments](https://www.unicode.org/reports/tr31/tr31-10.html#Specific_Character_Adjustments)" of UAX \# 31, starting in revision 10 (for Unicode version 5.2, 2009).

Summary
--

The end result _must_ be a static list of valid identifier characters that does _not_ "depend on the underlying platform for their Unicode behavior". It is perfectly acceptable for methods such as `CharUnicodeInfo.GetUnicodeCategory()` to reflect newer versions of Unicode upon updates to .NET. However, it is _not_ acceptable for any such changes to be reflected in which characters are valid in identifiers. Any changes in the future would need to be an updated static list combined with an updated specification that indicates the updated Unicode version being used.

For example, SQL Server uses the `ID_Start` and `ID_Continue` definitions from Unicode 3.2 (plus a few customizations, and minus any support for supplementary characters). This is stated in the documentation and has been consistent across at least 7 versions of SQL Server and updates to the underlying OS. While Unicode 3.2 is quite old and it would certainly be nice to have the definitions updated, it is at least an otherwise proper implementation.

Related Issues
--

* dotnet/csharpstandard#304
* dotnet/csharplang#1742
* https://github.com/dotnet/roslyn/issues/9731
* https://github.com/dotnet/roslyn/issues/13474
* https://github.com/dotnet/roslyn/issues/13560