microsoft / microsoft/language-server-protocol

Language Server Index Format: use a non-UTF-16 encoding as the default for `positionEncoding`

Open
#872 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

feature-request lsif
Dominant language
TypeScript
Stars
13k
Forks
1k
Avg merge
6d 1h
Merged PRs (30d)
10

Description

From the lengthy discussions in https://github.com/microsoft/language-server-protocol/issues/376 , a lot of language servers and non-JavaScript based clients have been inconvenienced due to the UTF-16 position counting. In Language Server Index Format, which does not have the backward compatibility burden, we should probably move away from UTF-16 which is considered inferior by a majority of developers. As the recommended default, both UTF-32 code points and UTF-8 bytes can work but it seems that code points are convenient for most implementations.

When opening a non UTF-8 encoded file, an implementation may likely represent it with a sequence of code points. I believe most implementations have convenient APIs to count code points but counting UTF-8 bytes may require more transcoding.

  /**
   * The string encoding used to compute line and character values in
   * positions and ranges. Currently only 'utf-16' is support due to the
   * limitations in LSP.
   */
  positionEncoding: 'utf-16',

A survey on the Position.character offsets encoding supported by the clients:

Language client Counting method of Position.character offsets  
eglot UTF-32 codepoints UTF-32 by default, optionally UTF-16
LanguageServer-neovim UTF-32 codepoints  
lsp-mode UTF-32 codepoints  
vim-lsp UTF-32 codepoints  
ycm UTF-16 code units Would prefer UTF-8, but only if standardized
VSCode UTF-16 code units  

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the linked LSP position-encoding discussion and the TypeScript definition for positionEncoding shown in the issue. Determine the agreed non-UTF-16 default and its compatibility implications; the work is done when the LSIF specification and related definition consistently document and use that default.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
api
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.