Introduce a `Language` type to provide consistent language information of strings.

Currently, Python has no consistent way to indicate when a programming language is represented as a string that the string follows the syntax of a particular programming language.

This means that languages represented as strings cannot be syntax highlighted, resulting in a significant loss of productivity, readability, and an increase in bugs and errors when dealing with other languages as strings.

This article gives an example of the current problem.

...

A component in my library is a combination of python code, html, css and javascript. Currently I glue things together with a python file, where you put the paths to the html, css and javascript. When run, it brings all of the files together into a component. But for small components, having to juggle four different files around is cumbersome, so I’ve started to look for a way to put everything related to the component in the same file. This makes it much easier to work on, understand, and with fewer places to make path errors.

Example:

class Calendar(component.Component):
    template_string = '<span class="calendar"></span>'
    css_string = '.calendar { background: pink }'
    js_string = 'document.getElementsByClassName("calendar)[0].onclick = function() { alert("click!") }'

Seems simple enough, right? The problem is: There’s no syntax highlighting in my code editor for the three other languages. This makes for a horrible developer experience, where you constantly have to hunt for characters inside of strings. You saw the missing quote in js_string right? 🙂

...

Traditional approaches and issues

Typical case

Typically, syntax highlighting is not provided at all because there is no way for the editor to know the language of the string, which leads to several drawbacks.

Batch syntax highlighting of raw strings for regexes in VSCode

VSCode provides simple syntax highlighting for regexes when using raw strings, as shown below.

However, this approach has several drawbacks. First of all, it doesn't generalize to languages other than regexes. Also, since raw strings aren't just for regexes, it creates a visual distraction for people who want to use raw strings for non-regex reasons, such as Windows paths.

Below is an example of syntax highlighting for regex applied to Windows path, which actually reduces readability.

Language and LiteralLanguage

Language is a subtype of str that indicates that the string represents a specific language. LiteralLanguage is a subtype of LiteralString, and is used in the same way as Language.

Language takes a single type argument, and in its place you put the name of the language, for example, Language["html"].

Editors should provide basic syntax highlighting for string literals set to types Language or LiteralLanguage. Consider code blocks in Markdown.

The Language type may also be implied by the type of the parameter.

from typing import Language

Language["html"] # The brackets hold the name of the language.

my_css: Language["css"] = "p { font-size: 20px; }" # This string is considered CSS and should be syntax highlighted.


def get_html(html: Language["html"]):
    ...

get_html("<p>hello, world!</p>") # This string is considered HTML and should be syntax highlighted.


def dreamberd_compiler(code: Language["java"]):
    ...

# `Language` can also be used in "reasonably similar code". This code should have syntax highlighting for Java.
dreamberd_compiler("var var hello = 123!")


def get_path(path: str):
    ...

def get_pattern(pattern: Language["re"]):
    ...

# Now it's not syntax highlighted as simply a raw string.
get_path(r"C:\Users\user\python.py") # This string shouldn't have any syntax highlighting.
get_pattern(r"a\rb+b?[abc]", "...") # This string should be syntax highlighted as a regex.

Errors

It is difficult to set the Language type to remain a Language type after an operation, as this would complicate the implementation and make it difficult to provide a clear criterion for the type.

For example, does Language["A"] + Language["A"] always result in Language["A"]? Of course it often does, but it's very hard to generalize.

The case of Language["A"] + Language["B"] is also tricky. Should we catch the type as Language["A"], or should it be Language["B"]? And what about Language["A"].strip()? It's hard to maintain consistency or a single standard for these operations. Therefore, Language should be considered more as a feature for annotation than for complex static type checking.

Therefore, a type checker should accept the target of a given Language type as legitimate if it is a string, regardless of its contents, and an editor should not raise an error if it fails to parse.

Developers should also not expect that when they accept a value annotated with `Language' that the string is fully valid code that will pass the language's compiler.

Conversely, Language can be used for code that is "reasonably close" to the appearance of the language. Developers should consider whether syntax highlighting helps or hinders users when deciding whether to use Language or just use str for languages that are not exactly the same as the target language.

Post-operation type

The type Language should be treated as str when computed, and LiteralLanguage should be treated as LiteralString when computed.

# In the case of `LiteralLanguage`

literal_html: LiteralLanguage["html"] = "<h1>Hello, world!</h1>"
literal_sql: LiteralLanguage["sql"] = "SELECT CustomerName, City FROM Customers;"
my_literal: LiteralString = "Contents: {}"

# When two different languages are synthesized, both variables are considered `LiteralString`.
reveal_type(literal_html + literal_sql)  # type: LiteralString

# If two strings of the same language are composited, or if `Language` is composited with a `Literal`, it should still be considered a `LiteralString`.
reveal_type(literal_html + literal_html)  # type: LiteralString
reveal_type(literal_html + " ")  # type: LiteralString
reveal_type(f"""
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
</head>
<body>
    {literal_html}
<body>
""")  # type: LiteralString

# A `LiteralString` can be cast to a `LiteralLanguage`.
template: LiteralLanguage["html"] = reveal_type(f"""
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
</head>
<body>
    {literal_html}
<body>
""")  # type: LiteralLanguage["html"]

# For all other operations, a `LiteralLanguage` is considered a `LiteralString`.
reveal_type(my_literal.format(literal_html))  # type: LiteralString
reveal_type(input() + literal_html)  # type: str

# In the case of `Language`

use_input = input().lower().startswith("y")
html: Language["html"] = "<h1>Hello, world!</h1>" if use_input else input()
literal_sql: LiteralLanguage["sql"] = "SELECT CustomerName, City FROM Customers;"
css: Language["css"] = "SELECT CustomerName, City FROM Customers;"
my_literal: LiteralString = "Contents: {}"

reveal_type(html + literal_sql)  # type: str
reveal_type(html + query)  # type: str

reveal_type(html + html)  # type: str
reveal_type(html + " ")  # type: str
reveal_type(f"""
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
</head>
<body>
    {html}
<body>
""")  # type: str

template: Language["html"] = reveal_type(f"""
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
</head>
<body>
    {literal_html}
<body>
""")  # type: Language["html"]

reveal_type(my_literal.format(html))  # type: str
reveal_type(input() + html)  # type: str

BytesLanguage?

ByteLanguage is the bytes version of Language. We should think about whether we need this type.

However, there is no type called LiteralBytes, so at least LiteralBytesLanguage can't exist.

Language names

The language identifier in Language must be lowercase, e.g. Language["python"] instead of Language["Python"].

For language names, it seems like a good idea to use what is used for code blocks in Markdown that developers are familiar with, but the exact definition of this is up to the editor.

Supported languages

A list of supported languages is beyond the scope of this documentation and should be up to each editor's implementation. However, editors should be able to provide basic syntax highlighting for common languages like Python, HTML, SQL, etc.

No implementation files, entry points, or tests are identified in the issue. Start by reviewing the Language, LiteralLanguage, post-operation type, BytesLanguage, and language-name sections, then resolve the open design choices; done means the type's semantics and supported editor behavior are specified consistently.