Proposal to add textDocument/ast

This is a proposal to add a textDocument/ast to the LSP protocol. The idea is to have a capability that returns an AST from a selection.

• The specification can specify naming and formatting conventions for various tokens and structures that are common across various language ASTs, to maximize code reuse, while allowing for any language AST to include any additional structures that are unique to the language. It would return the AST in JSON format.
• It can replace treesitter in a better way, since it's using the real parser of the language, rather than what treesitter does which is to reimplement the parser in an often ad-hoc, incomplete, and bug ridden way. For example, this capability could be used for:
  • Robust, effectively error prone, syntax highlighting.
  • Code folding.
  • Auto-indentation that never messes up.
  • Any other thing tree sitter does.
• In general, it can also be used for any sort of static analysis of the syntax of the code, which could be combined with other LSP calls to do really interesting static analysis on the editor side, to add really useful features in the future.

The following exist but don't actually result an AST:

• textDocument/documentSymbol
  • Provides a hierarchical representation of the symbols in a document, such as classes, methods, and variables. While this doesn't give you a full AST, it offers a tree-like structure representing the code’s high-level organization.
• textDocument/semanticTokens
  • Returns token-level semantic information about the document. This is useful for syntax highlighting and understanding the roles of various elements in the code but is not a direct representation of the AST.
• textDocument/definition and textDocument/references
  • These requests return locations in the code related to a symbol, providing context-aware navigation that relies on an understanding of the code's structure.

Start by reviewing the existing LSP request definitions, particularly textDocument/documentSymbol and textDocument/semanticTokens, and the specification's conventions for capabilities and JSON structures. Done would require an agreed scope and protocol design for textDocument/ast, including selection handling and AST naming and formatting conventions.