[Bug] monaco.editor.tokenize does not return correct results

Reproducible in vscode.dev or in VS Code Desktop?

• Not reproducible in vscode.dev or VS Code Desktop

Reproducible in the monaco editor playground?

• Not reproducible in the monaco editor playground

Monaco Editor Playground Code

monaco.editor.create(document.getElementById('container'), {
    value: "function hello() {\n\talert('Hello world!');\n}",
    language: 'typescript'
});

const tokens = monaco.editor.tokenize(`// My awesome TypeScript function!
export const areEqual = (a: number, b: number): boolean => {
    return a === b;
}`, "typescript");

console.log("tokens", tokens);

const tokensEscaped = monaco.editor.tokenize("// My awesome TypeScript function!\nexport const areEqual = (a: number, b: number): boolean => {\n\treturn a === b;\n}", "typescript");

console.log("tokensEscaped", tokensEscaped);

Reproduction Steps

1. Try to pass a preformatted string to monaco.editor.tokenize
2. Log out the results via console.log

Actual (Problematic) Behavior

Tokens are not detected (though lines are)

Expected Behavior

Tokens to be detected!

Additional Context

The strange thing is that if you paste the following into the terminal, it works:

tokens = monaco.editor.tokenize(`// My awesome TypeScript function!
    export const areEqual = (a: number, b: number): boolean => {
        return a === b;
    }`, "typescript");
console.log("tokens", tokens);
tokens (4) [Array(1), Array(27), Array(9), Array(1)]

I feel like this is some sort of initialization issue... but calling monaco.editor.tokenize even multiple times within my code doesn't seem to solve the problem, I still get empty token lines... very strange.

As you see I thought it was an escaping of \n and \t issue, but even the escaped single line string doesn't work....

Perhaps this is something small, but I can't see what is going wrong here. Need another set of eyes.

Start with the monaco.editor.tokenize calls in the Monaco Editor Playground reproduction and compare the preformatted template string with the escaped single-line string. Investigate why both produce lines without tokens while the terminal example works; done means the supplied TypeScript snippet returns detected tokens consistently.