microsoft / microsoft/language-server-protocol
Language Server Index Format: use a non-UTF-16 encoding as the default for `positionEncoding`
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 13k
- Forks
- 1k
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 10
Description
From the lengthy discussions in https://github.com/microsoft/language-server-protocol/issues/376 , a lot of language servers and non-JavaScript based clients have been inconvenienced due to the UTF-16 position counting. In Language Server Index Format, which does not have the backward compatibility burden, we should probably move away from UTF-16 which is considered inferior by a majority of developers. As the recommended default, both UTF-32 code points and UTF-8 bytes can work but it seems that code points are convenient for most implementations.
When opening a non UTF-8 encoded file, an implementation may likely represent it with a sequence of code points. I believe most implementations have convenient APIs to count code points but counting UTF-8 bytes may require more transcoding.
/**
* The string encoding used to compute line and character values in
* positions and ranges. Currently only 'utf-16' is support due to the
* limitations in LSP.
*/
positionEncoding: 'utf-16',
A survey on the Position.character offsets encoding supported by the clients:
| Language client | Counting method of Position.character offsets | |
|---|---|---|
| eglot | UTF-32 codepoints | UTF-32 by default, optionally UTF-16 |
| LanguageServer-neovim | UTF-32 codepoints | |
| lsp-mode | UTF-32 codepoints | |
| vim-lsp | UTF-32 codepoints | |
| ycm | UTF-16 code units | Would prefer UTF-8, but only if standardized |
| VSCode | UTF-16 code units |
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the linked LSP position-encoding discussion and the TypeScript definition for positionEncoding shown in the issue. Determine the agreed non-UTF-16 default and its compatibility implications; the work is done when the LSIF specification and related definition consistently document and use that default.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- api
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100