jgm / jgm/skylighting

How does `tokenize` separate `SourceLine`s?

Open
#182 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Haskell
Stars
218
Forks
67
PR merge metrics
No merged PRs in 30d

Description

The function `tokenize` outputs an array of `SourceLine`s. Is it guaranteed that if i provide it with `Text` with some multiline code, every line's tokens will be mapped to one `SourceLine` without any formatting/line changing behaviour?

For example, if I had this single line of oddly written piece of C code:
```c
int do_something() { call_a_function(); int some_number = 3; return 123; }
```
Would very token (e.g. `int`, `do_something`, `()`, `{`, `call_a_function` ...) will end up on the same `SourceLine` with every token in sequence?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.