dart-lang / dart-lang/language

Whitespace is underspecified in punctuation

Open
#2,735 0 comments 0 reactions 0 assignees View on GitHub
question specification technical-debt
Dominant language
TeX
Stars
2.9k
Forks
239
Avg merge
2d 18h
Merged PRs (30d)
14

Description

Cf. https://github.com/dart-lang/sdk/issues/50776.

The Dart grammar specifies symbol literals in terms of smaller parts, i.e., symbols are not recognized at the lexical level, but rather at the context free level. This implies that it is possible, according to the grammar, to have whitespace in symbol literals, for example `# foo . bar`.

The specification says that the value of the symbol is based on the source code of the symbol literal without any whitespace (which is again a confirmation that the whitespace is allowed).

A similar situation exists with a number of punctuation symbols which are used in different ways in different contexts (in particular `>` which can be the end of an actual type argument list, and a part of an operator like `>>` or `>>>=`).

However, the tools (the analyzer and the common front end) report an error for many occurrences of whitespace, including `#[ ]`, `#[] =`, `x >>> = 1`, etc.

A Dart parser needs to maintain the detailed structure during lexical analysis, because the same sequence of characters can have a different structure, depending on the context: `x >>>= 1` makes `>>>=` a single compound-assignment operator, but `List>>==List>` makes each of the `>` an end-of-type-arguments marker, and `==` an equality operator. So we can't simply let a lexer swallow `>>>=` as one token.

This implies that the rules that we might have to make `# [ ]` an error would be specified outside the grammar.

(Aside: As far as I know, the Dart parser used by the analyzer and the common front end does actually step out of the level of regular languages during the construction of tokens, e.g., by changing `>` `>` to `>>` if there are no matching `<`s in the tokens produced so far, but we can't assume that all Dart parsers will be able to match up parenthesis-like tokens during lexical analysis.)

It would make sense to specify such rules about errors for whitespace, at least in the cases where the tools currently report an error for this kind of whitespace.

Contributor guide

Open the contributing guide

Research direction

Start with the Dart grammar and the linked Dart SDK issue 50776, then compare the specification with the analyzer and common front end behavior for whitespace in symbol literals and punctuation operators. Define the specification rules for the cited cases, including when whitespace is an error, and ensure they agree with the tools' reported behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
dart
Domain
compilers
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.