[p5.js 2.0+ Bug Report]: splitTokens() returns [] for a caret delimiter and throws for a backslash

Most appropriate sub-area of p5.js?

Utilities

p5.js version

2.x main (4b096e2)

Actual vs expected behavior

splitTokens builds a regex character class from the delimiter characters with only [ and ] special-cased (the 2015 fix for #1004), so other regex metacharacters corrupt the class:

splitTokens('a^b', '^');   // actual []                       expected ['a', 'b']
                           // the ^ negates the class, so every character becomes a delimiter
splitTokens('a\b', '\'); // actual SyntaxError: Invalid regular expression: /[\]/g
                           // expected ['a', 'b']

A - between delimiter characters also silently forms a range instead of matching the literal characters.

Steps to reproduce

Outputs above are from executed runs against current main.

Note

I am aware splitTokens was deprecated in 2.0 (#7624), so feel free to close this as wontfix if the function is not receiving fixes anymore. If it is, I have a small fix ready (escape \, ], ^, - when building the class) with unit tests for both broken cases plus regression coverage for the existing bracket, array-delimiter and whitespace behaviors; the two new tests fail on current main and pass with the change. Will open the PR once approved.

Start at the splitTokens implementation and its existing unit tests. Reproduce the caret and backslash cases, then check coverage for bracket, array-delimiter, and whitespace behavior. Done means literal special-character delimiters produce the expected tokens without breaking the existing cases.