🧪 Tests: Add tests for remaining functions in tokens.ts

Bug Report Checklist

• I have tried restarting my IDE and the issue persists.
• I have pulled the latest main branch of the repository.
• I have searched for related issues and found none that matched my issue.

Overview

#6 added a first round of unit tests. Let's add some more tests! 🙌

The src/tokens.ts file is not yet tested. This issue tracks adding in tests for its exported functions.

Additional Info

See #6 for examples of tests: including those that work with AST nodes and/or type checking.

There might be existing tests in https://github.com/ajafff/tsutils that are relevant here. If you do find them, it's fine to copy & paste them over - as long as there's a comment like this above any area of copy & pasted code (or on top of a file if the entire file is copy & pasted):

// Code largely based on https://github.com/ajafff/tsutils
// Original license: https://github.com/ajafff/tsutils/blob/26b195358ec36d59f00333115aa3ffd9611ca78b/LICENSE

Start by reading the exported functions in src/tokens.ts and the test examples from issue #6. Check whether relevant tests exist in tsutils, preserving the stated attribution if code is copied. Done means the remaining exported token functions have unit tests covering applicable AST-node and type-checking behavior.