Proposal: Adding nullable reference type features to nullable value types (16.3, Core 3)

# Augmenting Nullable Value Types

The upcoming nullable reference types (NRTs) feature builds on the existing nullable *value* types (NVTs) for syntax and intuition, but differs in several ways.

One big difference is that NVTs have a different runtime representation that their nonnullable counterparts, whereas NRTs are indistinguishable from nonnullable ones at the runtime level, and are only differentiated at compile time, in source and metadata. This is a fundamental difference that is a result of deliberate design decisions, and can't really be remedied.

However, some of the novel aspects of NRTs might be "backported" to NVTs, diminishing the feature gap between them, and providing useful expressiveness.

# Tracking null state for NVTs

A key feature of NRTs is that we track null state for variables of reference type through a flow analysis, so that we can warn at points of dereference if the variable might be null:

``` c#
void M(Person? p)
{
WriteLine(p.Name); // warning: p might be null
if (p != null) { WriteLine(p.Name); } // No warning
WriteLine(p?.Name ?? ""); // No warning
if (p == null) return; WriteLine(p.Name); // No warning
}
```

Similarly, we could track null state for NVTs. The null state would come into play when boxing a NVT to a NRT, or when accessing the `Value` of a NVT:

``` c#
void M(int? i)
{
IComparable c = i; // Warning: i might be null
IComparable? n = i; // null state: n may be null
WriteLine(n.ToString()); // Warning: n might be null
if (i != null)
{
c = i; // No warning
n = i; // null state: n is not null
WriteLine(n.ToString()); // No warning
}
var n = (IComparable)i; // Warning: casting away nullness
int x = i.Value; // Warning: i might be null
x = (int)i; // Warning: casting away nullness
}
```

Just like with NRTs, `!` can be used to suppress warnings, and to change the null state of a nullable value:

``` c#
void M(int? i)
{
WriteLine(i!.Value); // No warning
int x = i!; // No warning, null state: x is not null
}
```

## Special considerations for NVTs

The analysis should recognize null checks using `HasValue` as well as ones involving the `null` literal (which are generally translated to uses of `HasValue` by the compiler).

``` c#
int M(int? i)
{
if (i.HasValue) return i.Value; // No warning
else return -1;
}
```

Also, the analysis should account for the semantics of "lifted" operators. For every operator (intrinsic or user-defined) over non-nullable value types, there's a corresponding language-provided lifted operator, that works over the corresponding NVTs. The lifted operator returns null if either operand is null.

``` c#
int? a, b;
(a, b) = (7, 9);
int x = a + b; // no warning; the null state of the + result is not null
a = null;
x = a + b; // warning on assignment
```

**Pros:**
- The benefit of warnings on null-unsafe code is extended to NVTs

**Cons:**
- This might encourage use of `Value` over uses of `GetValueOrDefault` which is more efficient.

# Using nullable values as nonnullable values

Because of the null state tracking, NRTs allow use of the "underlying" value when the flow analysis says that the nullable reference isn't null.

NVTs, on the other hand, are a completely separate value, and even when you just checked for null, you still cannot use them as the underlying value directly; you have to get at it first with `.Value` or `.GetValueOrDefault()` or a cast.

If we do null state tracking for NVTs as proposed above, could we allow direct use as the underlying value when a nullable value is known not to be null?

One immediate obstacle is that NVTs are types in their own right, with their own members and a separate type identity wrt. overload resolution. For member access and conversions, they already have semantics, and any new semantics where they "pose" as the underlying type would have to be strictly *additional* and non-breaking. So they would kick in only in places where you'd get an error today.

For simplicity we should probably allow the additional member accesses and conversions regardless of whether the value is null or not, but then warn on it when they are applied to a nullable value that has the "may be null" null state. Allowing it regardless of null state also helps maintain the design principle from NRTs that the null state should never affect semantics, only whether warnings are yielded.

## Member access

The proposal is that instance members of the underlying type become available on the nullable type, with a warning when the value "may be null".

For back compat, members that are defined on `Nullable` should always shadow corresponding members on the underlying type. That is a pretty short list, though, and many of them (`ToString`, `Equals`, `GetHashCode`) work by calling through to the underlying type when the value is non-null, so only very few members of the underlying value would be effectively shadowed in the sense that the behavior is different. Those are members directly implementing the contract of the NVT (`Value`, `HasValue`, `GetValueOrDefault`), as well as reflection (`GetType`), which must acknowledge that a nullable value is different at runtime from its underlying value.

Other than those, members of the underlying type could be offered, and would imply an implicit indirection through the `Value` property.

``` c#
void M(int? i)
{
string s = i.ToString(); // int?.ToString
int x = i.CompareTo(7); // int.CompareTo + warning
if (i != null)
{
s = i.ToString(); // STILL int?.ToString
x = i.CompareTo(7); // no warning
}
}
```

**Pros:**
- Narrow the experience gap between NRTs and NVTs
- Help "hide" the separate-value-ness of NVTs

**Cons:**
- You're encouraged to rely on nullable tracking and check for null less, possibly leading to more exceptions when analysis is wrong
- Would have to translate into use of `Value`, even though `GetValueOrDefault` is more efficient when you're sure it's never null
- Between pattern matching (`if (i is int x)`) and null-conditionals (`s = i?.CompareTo(7);`) there are reasonable alternatives

## Conversion and overload resolution

Ideally we would straightforwardly allow a NVT to be implicitly converted to its underlying value type, with a warning if it "may be null". However, that would be a big breaking change, since it would make new overloads applicable, leading to ambiguities or silent changes of behavior. In today's betterness algorithm, non-nullable wins over nullable

Instead we'd need sort of a "Hail Mary" pass, where *if* overload resolution/assignment would otherwise fail, we add these conversions and try again. Thus, you'd get the following behavior:

``` c#
void N(int x);
void O(int? x); // 1
void O(int x); // 2

void M(int? n, int i)
{
N(n); // warning
N(i); // fine
O(n); // overload 1
O(i); // overload 2
if (n != null)
{
N(n); // no warning
O(n); // STILL overload 1
O((int)n); // overload 2
}
}
```

**Pros:**
- You get to treat NVTs as their underlying types, with warnings when null is a (recognized) danger

**Cons:**
- A separate pass is a big hammer, and gets super complicated in the language and compiler, and to the user
- This is different from how we handle operators, which get applied to NVTs through lifting

## Alternative: Lifting

There is already a language-level approach to making NVTs work smoothly for operators: lifting. For each operator over non-null value types, there is automatically a corresponding one over the corresponding NVTs (unless one already exists): What it does is to yield null if either operand is null, and the result of the underlying operator otherwise.

This is similar in functionality and typing to how the `?.` operator works with respect to a NVT receiver. In a sense, `?.` is an explicit lifting of the `.` operator.

Could we address the scenario by doing implicit lifting in more scenarios? It would look something like this:
- For a NVT `S?` we lift all the members of `S` to `S?` (except the ones that are already there), and return null if the receiver is null
- For every method overload `M` that takes at least one parameter of non-nullable value type , we introduce an overload where all non-nullable value types in the signature are replaced with their corresponding NVTs

For members, this really just means making `?.` implicit on NVTs.

FOr methods, the benefit is that the old overloads would be "better" than the new lifted ones, because non-nullable types in the signature are better than nullable ones. So existing methods would continue to bind the same way.

Mostly. There are still some breaking changes possible. For instance, if there is an overload with a *reference* type, then it could get ambiguous with a lifted NVT overload on the `null` literal:

``` c#
M(int i);
M(string s);

M(null); // ambiguous with M(int?) now?
```

We would need a tie breaker rule to make lifted methods take a backseat to non-lifted ones. That's doable.

Worse, though, there are still cases where we'd pick a *different* overload than before:

``` c#
M(int i);
M(object? o);

M(null); // M(int?) instead of M(object?) now?
```

The issue is that the NVT in the lifted method may still be better than a reference type in an existing overload.

In short, lifted methods still need to be treated specially, taking more of a backseat in overload resolution. Lifting, then, still requires a special "second phase", but would use "extra overloads" instead of the "extra conversions" proposed above.

**Pros:**
- "Less special casing" than the above proposal
- More similar to the present handling of operators

**Cons:**
- Still requires a special phase
- Would lead to an implicit chaining of null checks (if null then null) rather than early exit on null, which would hurt performance and eventually flag the problem 'at the end' rather than on the first undesired null.
- Doesn't really make good use of null tracking; just propagates nulls
- Doesn't help with direct conversion, as in `int i = n;`

# Recommendations

I would like to see us do null-tracking for NVTs. I don't think the downsides are significant.

I wish we could allow NVTs to be used as their underlying types, with a warning when they might be null. Doing this for member access is significantly less complicated than for conversions, but it would also seem inconsistent to only do the easy part.

I would like us to drill more on how best to solve the conversion case. If we can come up with something relatively elegant, without sacrificing back compat, then I think the whole package may be well worth considering. Otherwise I'd probably leave all of the new semantics for another day, and just do the null tracking.

LDM history:
- https://github.com/dotnet/csharplang/blob/master/meetings/2018/LDM-2018-09-19.md#add-nullable-reference-type-features-to-nullable-value-types
- https://github.com/dotnet/csharplang/blob/master/meetings/2018/LDM-2018-10-24.md#tracking-null-state-for-nullablet