Conditional + annotation causes erroneous typing

TypeScript Version: 33.9.4 & 4.0.0@beta

Search Terms: Type annotation wrongly overrides generic with errors

Code

const createMatrix = <D extends number, T>(
    dimensions: D,
    initialValues: T | null = null
): Matrix<D, T> => {
    const currentDimensionLength = dimensions;
    const remainingDimensions = dimensions - 1;
    const needsRecursion = remainingDimensions > 0;

    const currentMatrix = Array(currentDimensionLength).fill(initialValues);

    const finalMatrix = needsRecursion
        ? currentMatrix.map(() =>
              createMatrix(remainingDimensions, initialValues)
          )
        : currentMatrix;

    return finalMatrix as Matrix<D, T>;
};

type Matrix<D extends number, T> = D extends 1
    ? T[]
    : D extends 2
    ? T[][]
    : D extends 3
    ? T[][][]
    : any[][][][];

Expected behavior:
Adding type annotations to variables should be compatible with the functions return signature, causing no type errors

Actual behavior:
Assigning an annotated variable to the return value of the function causes an error. The generic T is wrongly inferred to be any of array where depth < D where it should be inferred to only the base type of the nested array. Removing the annotation solves the problem but removes type safety when variable assignment happens after creation.

The problem seems to be that (when annotated) the type is inferred to be the union of each type in Matrix that's preceding the conditional that returns true.

const n1: number[] = createMatrix(1); // <- works - generic T is set to `number`
const n2: number[][] = createMatrix(2); // <- doesn't work - generic T is set to `number | number[]`
const n3: number[][][] = createMatrix(3); // <- doesn't work - generic T is set to `number | number[] | number[][]`

const n4 = createMatrix(2); // <- regular inference works - generic T is `unknown`, return type is unknown[][]
const n5: number[][] = createMatrix(2, 0); // <- works when - initialValues is set - generic T is `number`, return type is number[][]
const n6: unknown[][] = createMatrix(2); // <- works fine, as type T is unknown. A more specific annotation should constrain `unknown` to `number`

Playground Link: Link

Related Issues:
None

Use the linked Playground reproduction with TypeScript 3.9.4 and 4.0.0-beta, focusing on the annotated n2 and n3 calls versus the unannotated and explicitly initialized cases. Trace the type inference and conditional return-type behavior until the source of the accumulated union is identified; done means the annotated calls produce the expected nested array types without errors.