dart-lang / dart-lang/language
Escaped reserved words
- Dominant language
- TeX
- Stars
- 2.9k
- Forks
- 239
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 14
Description
This is a proposed solution for #270:
Evolving a programming language is always challenging in a number of ways. Users often want new features, but those features need syntax and adding syntax without breaking existing programs is difficult.
In particular, it is impossible to add new reserved words to the language. A reserved word, by definition cannot be used by users as an identifier. This means that if, say, Dart 2.x turns `foo` into a reserved word, then any existing program using `foo` as a variable name, type name, member name, import prefix, etc. breaks with a syntax error.
The typical way Dart and other languages avoid this is by never adding new reserved words. Instead, they add "contextual keywords" or "built-in identifiers". These are identifiers that behave like keywords in some contexts but can be used by users as normal identifiers in other places.
For example, `show` behaves like a keyword when used after an import or export directive:
```dart
import 'foo.dart' show something;
```
But it can also be used as an identifier:
```dart
class Widget {
void show() { print("I am visible now."); }
}
```
In addition to not breaking existing programs, this has another advantages:
* **Natural-sounding words can be used without taking them away from users.** It would be a shame if a language designed for user interfaces didn't let users create methods named `show` and `hide`. It would be annoying if a language frequently used for web apps couldn't use `get`. It would be really strange if a language that had a [Set][] type in its core library didn't let you name a variable `set`.
[set]: https://api.dartlang.org/stable/2.2.0/dart-core/Set-class.html
But it carries a number of disadvantages:
* **Syntax highlighting is harder.** A tool doesn't know whether to treat `await` as a keyword unless it tracks whether or not it is currently inside an async function. Simple tools like syntax highlighters rarely carry that context, so writing correct highlighters for Dart is challenging.
In practice, most just always highlight contextual keywords as if they were reserved words. That in turn encourages users to believe these *are* reserved, which then causes confusion when they stumble into code that uses it like an identifier.
* **Error recovery is harder.** Contextual keywords are almost always used for their keyword behavior and rarely as identifiers. When a user inadvertently uses the keyword in a place where it doesn't behave like a keyword, they usually want error messaging that tells them *why* it's not a keyword. Using `await` in a function that you forgot to mark `async` is the classic example.
But, because the keyword could technically be used as a identifier there too, it's hard for tools like IDEs to know what to tell users to fix their code. This means folks working on IDEs spend a lot more effort to deliver decent error messages than they would need to if the keyword was completely reserved.
* **Defining the context such that the keyword's use isn't ambiguous is harder.** The nice thing about a reserved word is that you know what it means regardless of where it appears in the user's code. But when the same lexeme can be used as both a keyword and an identifier, the grammar needs to be carefully designed to ensure those two cases don't collide.
For example, one annoyance of `await` is that has low precedence and doesn't chain nicely in method calls. Hard to read code like this is common:
```dart
await (await (await foo).bar).baz;
```
An obvious solution would be a postfix await syntax:
```dart
foo.await.bar.await.baz.await;
```
But this doesn't work since `await` here appears in a place where it could also be an identifier.
This makes it harder to evolve the language since any new syntax has to avoid these ambiguous cases.
* **Making semicolons optional is harder.** The ambiguity problem becomes acute when we try to make semicolons optional. For that to work gracefully, we need to ignore newlines in places where they obviously aren't meaningful, but "obvious" gets murky around contextual keywords.
Consider:
```dart
import 'foo.dart'
hide bar
```
Without an explicit semicolon separating the import directive from the next declaration, we have to decide whether to treat this as:
```dart
import 'foo.dart' hide bar;
```
Or:
```dart
import 'foo.dart';
hide bar; // A variable "bar" of type "hide".
```
Because `hide` is a contextual keyword, both are plausible. There are similar ambiguities around `Function`, `await`, `async`, etc.
* **It's just harder to understand.** Many programmers don't know "contextual keywords" even exist. They have a simpler mental model that a given name is either completely reserved and owned by the language or not meaningful to the language at all. They read code assuming this mental model, which works correctly most of the time, and then are very confused when they run into places where it breaks down.
Having contextual keywords increases the cognitive load of the language, especially given that Dart actually has several categories of contextual keywords, each with their own special rules.
In other words, contextual keywords make the language bigger, more confusing, and harder to change. They technically preserve compatibility, but with a high tax.
## A Model for Evolving the Language
In the past, most programming languages evolved with a policy of 100% backwards compatibility. That's great for, well, compatibility, but the trade-off is that the language gets monotonically more complex over time.
The increasing complexity means people now avoid C++ completely because it's simply too large for a new user to learn. If you didn't get on the C++ train a decade ago, it's very difficult to catch up. (See [1][], [2][].)
[1]: https://www.theregister.co.uk/2018/06/18/bjarne_stroustrup_c_plus_plus/
[2]: http://aras-p.info/blog/2018/12/28/Modern-C-Lamentations/
[3]: https://blog.tartanllama.xyz/initialization-is-bonkers/
The other problem is that language features have to be compromised from their ideal form in the name of compatibility. For example, if we wanted to add non-nullable types in a non-breaking way, then we'd have to treat every existing type annotation as nullable, since that's what they mean today.
In order to get a non-nullable type, you'd need some explicit marker like `!`. But that's the wrong default. Empirical analysis shows something like 90% of variables are non-nullable, so forcing users to opt *in* to that only the majority of their types is a strictly worse feature.
To avoid that, Dart, Rust, and other languages are moving to a model where compatibility is preserved through a combination of opting in to new features and migration tooling. Requiring an opt in means existing code continues to work as it does today.
At the point that you opt in, you can also run a tool that changes your *existing* code to get it to a form that makes the most sense in the context of the new feature. With non-nullable types, that lets us make non-nullable the default, leading to cleaner code post-migration without piles of pointless `!`. It's even theoretically possible to have migrations that purely *remove* deprecated features, giving us a way to simplify the language over time by removing functionality that no longer carries its weight.
This model generally works well for syntax changes, but one area where it breaks down is when the migration tooling would change the public API of a library. At that point, a user can't freely opt in to the change because it forces them to break *their* existing users.
An example that gets to the point of this proposal is reserving a new word. Let's say we want to turn `async` into a fully reserved word. We could write a tool that found any existing uses of `async` as an identifier and re-wrote them to something like `myAsync`. The resulting code now no longer has syntax errors. But if those identifiers are in public members, any library importing the migrated one are broken. In other words, migration isn't encapsulated.
## Escaping Reserved Words
This proposal solves that for reserved words by providing a syntax that lets you explicitly use *any* reserved word (new or old) as an identifier. We [borrow a feature from Swift][swift] and allow a backticks around any reserved word or identifier:
```dart
var `for` = "a variable named 'for'";
```
[swift]: https://swift.unicorn.tv/articles/reserved-words-in-swift-and-how-to-escape-them
This provides two main benefits:
* **We can add new reserved words.** In order to do so, we require an opt in and then ship a tool that finds any existing uses of the keyword and wraps them in backticks. This gets the code back to its original meaning without changing its public API.
(Eventually, a library author will probably want to stop using the now-reserved word in their API, but they can do that at their discretion.)
* **We can gracefully interop with other languages and systems that use Dart reserved words as identifiers.** When generating Dart APIs that interop with JavaScript, protobufs, JSON, etc. you can provide access to identifiers in those other systems even if they happen to be a reserved word.
We have a general goal of making the language easier to evolve, and this feature would give us one small mechanism to let us evolve the set of keywords in a mechanically-migratable way.
Contributor guide
Assessment
This issue has not been assessed yet.