SemanticTokens request returns empty output

Rust-Analyzer Version

a28077b2 2023-12-04

Rustc Version

rustc 1.74.0 (79e9716c9 2023-11-13) (Homebrew)

Description

I am trying to query semantic tokens for rust source files from the CLI using rust-analyzer. I have written the following Python script to initialize the server and then fire JSON RPC requests to the server's STDIN.

import json
import subprocess
import time
import os
import sys


print("Start the language server")
lsp_process = subprocess.Popen(
    ["rust-analyzer"],
    stdin=subprocess.PIPE,
    stdout=sys.stdout,
    stderr=sys.stderr
)

def write_message(lsp_process, message):
    message_json = json.dumps(message).encode('utf-8')
    print("\n=============================\n")
    print("Sending:\n")
    print(f"Content-Length: {len(message_json)}\r\n\r\n".encode('utf-8') + message_json)
    print("\n=============================\n")
    lsp_process.stdin.write(f"Content-Length: {len(message_json)}\r\n\r\n".encode('utf-8') + message_json)
    lsp_process.stdin.flush()

init_message = {
    "jsonrpc": "2.0",
    "method": "initialize",
    "id": 1,
    "params": {
        "rootUri": f"file://{os.getcwd()}/Rust/Tests/",
        "capabilities": {},
        "initializationOptions": {
            "semanticTokens": True
        }
    },
}
write_message(lsp_process, init_message)
time.sleep(5)

initialized_notification = {
    "jsonrpc": "2.0",
    "method": "initialized",
    "params": {}
}
write_message(lsp_process, initialized_notification)
time.sleep(5)

document_open_notification = {
    "jsonrpc": "2.0",
    "method": "textDocument/didOpen",
    "params": {
        "textDocument": {
            "uri": f"file://{os.getcwd()}/Rust/Tests/test2.rs",
            "languageId": "rust",
            "version": 1,
            "text": open(f"{os.getcwd()}/Rust/Tests/test2.rs").read()
        }
    }
}
write_message(lsp_process, document_open_notification)
time.sleep(20)

semantic_tokens_request = {
    "jsonrpc": "2.0",
    "id": 2,
    "method": "textDocument/semanticTokens/full",
    "params": {
        "textDocument": {
            "uri": f"file://{os.getcwd()}/Rust/Tests/test2.rs"
        }
    }
}
write_message(lsp_process, semantic_tokens_request)
time.sleep(10)

document_close_notification = {
    "jsonrpc": "2.0",
    "method": "textDocument/didClose",
    "params": {
        "textDocument": {
            "uri": f"file://{os.getcwd()}/Rust/Tests/test2.rs"
        }
    }
}
write_message(lsp_process, document_close_notification)
time.sleep(2)

lsp_process.terminate()

The file test2.rs is copied from here (just to convince the reader that it shouldn't lack semantic tokens).

However, executing the above gives the following output, with the response for the textDocument/semanticTokens/full request being an empty array.

Start the language server

=============================

Sending:

b'Content-Length: 207\r\n\r\n{"jsonrpc": "2.0", "method": "initialize", "id": 1, "params": {"rootUri": "file:///Users/ineilpaul/Downloads/Semantic_LSP/Rust/Tests/", "capabilities": {}, "initializationOptions": {"semanticTokens": true}}}'

=============================

Content-Length: 2538

{"jsonrpc":"2.0","id":1,"result":{"capabilities":{"positionEncoding":"utf-16","textDocumentSync":{"openClose":true,"change":2,"save":{}},"selectionRangeProvider":true,"hoverProvider":true,"completionProvider":{"triggerCharacters":[":",".","'","("],"completionItem":{"labelDetailsSupport":false}},"signatureHelpProvider":{"triggerCharacters":["(",",","<"]},"definitionProvider":true,"typeDefinitionProvider":true,"implementationProvider":true,"referencesProvider":true,"documentHighlightProvider":true,"documentSymbolProvider":true,"workspaceSymbolProvider":true,"codeActionProvider":true,"codeLensProvider":{"resolveProvider":true},"documentFormattingProvider":true,"documentRangeFormattingProvider":false,"documentOnTypeFormattingProvider":{"firstTriggerCharacter":"=","moreTriggerCharacter":[".",">","{","("]},"renameProvider":{"prepareProvider":true},"foldingRangeProvider":true,"declarationProvider":true,"workspace":{"workspaceFolders":{"supported":true,"changeNotifications":true},"fileOperations":{"willRename":{"filters":[{"scheme":"file","pattern":{"glob":"**/*.rs","matches":"file"}},{"scheme":"file","pattern":{"glob":"**","matches":"folder"}}]}}},"callHierarchyProvider":true,"semanticTokensProvider":{"legend":{"tokenTypes":["comment","decorator","enumMember","enum","function","interface","keyword","macro","method","namespace","number","operator","parameter","property","string","struct","typeParameter","variable","angle","arithmetic","attribute","attributeBracket","bitwise","boolean","brace","bracket","builtinAttribute","builtinType","character","colon","comma","comparison","constParameter","derive","deriveHelper","dot","escapeSequence","invalidEscapeSequence","formatSpecifier","generic","label","lifetime","logical","macroBang","parenthesis","punctuation","selfKeyword","selfTypeKeyword","semicolon","typeAlias","toolModule","union","unresolvedReference"],"tokenModifiers":["documentation","declaration","static","defaultLibrary","async","attribute","callable","constant","consuming","controlFlow","crateRoot","injected","intraDocLink","library","macro","mutable","public","reference","trait","unsafe"]},"range":true,"full":{"delta":true}},"inlayHintProvider":{"resolveProvider":true},"experimental":{"externalDocs":true,"hoverRange":true,"joinLines":true,"matchingBrace":true,"moveItem":true,"onEnter":true,"openCargoToml":true,"parentModule":true,"runnables":{"kinds":["cargo"]},"ssr":true,"workspaceSymbolScopeKindFiltering":true}},"serverInfo":{"name":"rust-analyzer","version":"1.74.1 (a28077b2 2023-12-04)"}}}
=============================

Sending:

b'Content-Length: 57\r\n\r\n{"jsonrpc": "2.0", "method": "initialized", "params": {}}'

=============================


=============================

Sending:

b'Content-Length: 23753\r\n\r\n{"jsonrpc": "2.0", "method": "textDocument/didOpen", "params": {"textDocument": {"uri": "file:///Users/ineilpaul/Downloads/Semantic_LSP/Rust/Tests/test2.rs", "languageId": "rust", "version": 1, "text": "use rand::distributions::WeightedIndex;\\nuse rand::prelude::*;\\nuse std::cell::RefCell;\\nuse std::cmp::{min, Ordering};\\nuse std::collections::BinaryHeap;\\nuse std::rc::Rc;\\n\\ntype NodeRef = Rc<RefCell<Node>>;\\ntype HypothesisRef = Rc<RefCell<Hypothesis>>;\\ntype Agenda = BinaryHeap<Hypothesis>;\\n\\nstruct Hypothesis {\\n    node_ref: NodeRef,\\n    next: Option<HypothesisRef>,\\n    fx: f64,\\n    gx: f64,\\n}\\nimpl Hypothesis {\\n    pub fn new(node_ref: NodeRef, next: Option<HypothesisRef>, fx: f64, gx: f64) -> Self {\\n        Self {\\n            node_ref,\\n            next,\\n            fx,\\n            gx,\\n        }\\n    }\\n}\\nimpl PartialEq for Hypothesis {\\n    fn eq(&self, other: &Self) -> bool {\\n        self.fx == other.fx\\n    }\\n}\\nimpl Eq for Hypothesis {}\\nimpl PartialOrd for Hypothesis {\\n    fn partial_cmp(&self, other: &Self) -> Option<Ordering> {\\n        Some(self.cmp(other))\\n    }\\n}\\n// TODO Maybe use Ordered Floats (https://docs.rs/ordered-float/1.0.2/ordered_float/)\\nimpl Ord for Hypothesis {\\n    fn cmp(&self, other: &Self) -> Ordering {\\n        if self.fx < other.fx {\\n            Ordering::Less\\n        } else {\\n            Ordering::Greater\\n        }\\n    }\\n}\\n\\n/// Structure to implement Viterbi algorithm to find the best encoding, or sample\\n/// from all possible encodings of a given sentence.\\n#[derive(Debug)]\\npub struct Lattice<\'a> {\\n    pub(super) sentence: &\'a str,\\n    len: usize,\\n    nodes: Vec<NodeRef>,\\n    pub(super) begin_nodes: Vec<Vec<NodeRef>>,\\n    pub(super) end_nodes: Vec<Vec<NodeRef>>,\\n    _bos_id: usize,\\n    _eos_id: usize,\\n}\\n\\nimpl std::fmt::Display for Lattice<\'_> {\\n    fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {\\n        let display_pieces = |nodes: &Vec<Vec<NodeRef>>| {\\n            nodes\\n                .iter()\\n                .map(|l| {\\n                    l.iter()\\n                        .map(|n| self.piece(&n.borrow()))\\n                        .collect::<Vec<_>>()\\n                })\\n                .collect::<Vec<_>>()\\n        };\\n\\n        f.debug_struct(\\"Lattice\\")\\n            .field(\\"sentence\\", &self.sentence)\\n            .field(\\"begin_nodes\\", &display_pieces(&self.begin_nodes))\\n            .field(\\"end_nodes\\", &display_pieces(&self.end_nodes))\\n            .finish()\\n    }\\n}\\n\\n/// A node from the lattice, that helps reconstruct the underlying `String`\\n#[derive(Debug, Clone)]\\npub struct Node {\\n    // Vocabulary id\\n    pub(super) id: usize,\\n    // Local lattice identifier\\n    pub(super) node_id: usize,\\n    pos: usize,\\n    length: usize,\\n    prev: Option<NodeRef>,\\n    backtrace_score: f64,\\n    score: f64,\\n}\\n\\nimpl PartialEq for Node {\\n    fn eq(&self, other: &Node) -> bool {\\n        self.id == other.id\\n    }\\n}\\n\\nimpl Node {\\n    pub fn new(id: usize, node_id: usize, pos: usize, length: usize, score: f64) -> Self {\\n        Self {\\n            id,\\n            node_id,\\n            pos,\\n            length,\\n            prev: None,\\n            score,\\n            backtrace_score: 0.0,\\n        }\\n    }\\n}\\n\\n/// Returns log(exp(x) + exp(y)).\\n/// if init_mode is true, returns log(exp(y)) == y.\\n/// log(\\\\sum_i exp(a[i])) can be computed as\\n/// for (int i = 0; i < a.size(); ++i)\\n///   x = LogSumExp(x, a[i], i == 0);\\nfn log_sum_exp(x: f64, y: f64, init_mode: bool) -> f64 {\\n    if init_mode {\\n        y\\n    } else {\\n        let (vmin, vmax) = if x > y { (y, x) } else { (x, y) };\\n        let k_minus_log_epsilon = 50.0;\\n        if vmax > vmin + k_minus_log_epsilon {\\n            vmax\\n        } else {\\n            vmax + ((vmin - vmax).exp() + 1.0).ln()\\n        }\\n    }\\n}\\n\\nimpl<\'a> Lattice<\'a> {\\n    pub fn from(sentence: &\'a str, bos_id: usize, eos_id: usize) -> Self {\\n        let len = sentence.len();\\n        let k_reserved_node_size = 16;\\n        // We are adding 2 tokens, bos and eos\\n        let mut nodes: Vec<NodeRef> = Vec::with_capacity(k_reserved_node_size);\\n        let mut begin_nodes = vec![Vec::with_capacity(k_reserved_node_size); len + 1];\\n        let mut end_nodes = vec![Vec::with_capacity(k_reserved_node_size); len + 1];\\n\\n        let bos = Rc::new(RefCell::new(Node::new(bos_id, 0, 0, 0, 0.0)));\\n        let eos = Rc::new(RefCell::new(Node::new(eos_id, 1, len, 0, 0.0)));\\n\\n        begin_nodes[len].push(Rc::clone(&eos));\\n        end_nodes[0].push(Rc::clone(&bos));\\n\\n        nodes.push(bos);\\n        nodes.push(eos);\\n\\n        Self {\\n            sentence,\\n            len,\\n            nodes,\\n            begin_nodes,\\n            end_nodes,\\n            _bos_id: bos_id,\\n            _eos_id: eos_id,\\n        }\\n    }\\n\\n    pub fn insert(&mut self, pos: usize, length: usize, score: f64, id: usize) {\\n        let node_id = self.nodes.len();\\n        let node = Rc::new(RefCell::new(Node::new(id, node_id, pos, length, score)));\\n\\n        self.begin_nodes[pos].push(Rc::clone(&node));\\n        self.end_nodes[pos + length].push(Rc::clone(&node));\\n\\n        self.nodes.push(node);\\n    }\\n\\n    pub fn viterbi(&mut self) -> Vec<NodeRef> {\\n        let len = self.len;\\n        let mut pos = 0;\\n        while pos <= len {\\n            if self.begin_nodes[pos].is_empty() {\\n                return vec![];\\n            }\\n            for rnode in &self.begin_nodes[pos] {\\n                rnode.borrow_mut().prev = None;\\n                let mut best_score = 0.0;\\n                let mut best_node: Option<NodeRef> = None;\\n                for lnode in &self.end_nodes[pos] {\\n                    let score = lnode.borrow().backtrace_score + rnode.borrow().score;\\n                    if best_node.is_none() || score > best_score {\\n                        // TODO can we remove this clone ?\\n                        best_node = Some(lnode.clone());\\n                        best_score = score\\n                    }\\n                }\\n                match best_node {\\n                    Some(bnode) => {\\n                        rnode.borrow_mut().prev = Some(Rc::clone(&bnode));\\n                        rnode.borrow_mut().backtrace_score = best_score;\\n                    }\\n                    None => return vec![],\\n                }\\n            }\\n            if let Some(c) = self.sentence[pos..].chars().next() {\\n                pos += c.len_utf8();\\n            } else {\\n                break;\\n            }\\n        }\\n\\n        let mut results: Vec<NodeRef> = vec![];\\n        let root = self.begin_nodes[len][0].borrow();\\n        let prev = root.prev.as_ref();\\n        if prev.is_none() {\\n            return vec![];\\n        }\\n        let mut node: NodeRef = prev.unwrap().clone();\\n        while node.borrow().prev.is_some() {\\n            results.push(node.clone());\\n            let n = node.borrow().clone();\\n            node = n.prev.as_ref().unwrap().clone();\\n        }\\n        results.reverse();\\n        results\\n    }\\n\\n    pub fn piece(&self, node: &Node) -> String {\\n        self.sentence[node.pos..node.pos + node.length].to_owned()\\n    }\\n\\n    pub fn tokens(&mut self) -> Vec<String> {\\n        self.viterbi()\\n            .iter()\\n            .map(|node| self.piece(&node.borrow()))\\n            .collect()\\n    }\\n\\n    pub fn nbest(&mut self, n: usize) -> Vec<Vec<NodeRef>> {\\n        match n {\\n            0 => vec![],\\n            1 => vec![self.viterbi()],\\n            _ => {\\n                // let k_reserved_hypothesis_size = 512;\\n                let mut agenda: Agenda = BinaryHeap::new();\\n                let mut hypotheses: Vec<Vec<NodeRef>> = vec![];\\n                let eos = self.eos_node();\\n                let score = eos.borrow().score;\\n                let hypo = Hypothesis::new(eos, None, score, score);\\n                agenda.push(hypo);\\n\\n                // Fill backtrace scores\\n                self.viterbi();\\n\\n                while !agenda.is_empty() {\\n                    let top = Rc::new(RefCell::new(agenda.pop().unwrap()));\\n                    let node = Rc::clone(&top.borrow().node_ref);\\n                    if node.borrow().id == self.bos_node().borrow().id {\\n                        let mut hypothesis = vec![];\\n                        let mut next: HypothesisRef =\\n                            Rc::clone(top.borrow().next.as_ref().unwrap());\\n                        while next.borrow().next.is_some() {\\n                            hypothesis.push(next.borrow().node_ref.clone());\\n                            let c: HypothesisRef = next.clone();\\n                            // let c: Ref<Hypothesis> = next.clone().borrow();\\n                            next = Rc::clone(c.borrow().next.as_ref().unwrap());\\n                        }\\n                        hypotheses.push(hypothesis);\\n                        if hypotheses.len() == n {\\n                            return hypotheses;\\n                        }\\n                    } else {\\n                        for lnode in &self.end_nodes[node.borrow().pos] {\\n                            let top_gx = top.borrow().gx;\\n                            let fx = lnode.borrow().backtrace_score + top_gx;\\n                            let gx = lnode.borrow().score + top_gx;\\n                            let hyp =\\n                                Hypothesis::new(Rc::clone(lnode), Some(Rc::clone(&top)), fx, gx);\\n                            agenda.push(hyp);\\n                        }\\n                        // When the input is too long or contains duplicated phrases,\\n                        // `agenda` will get extremely big. Here we avoid this case by\\n                        // dynamically shrinking the agenda.\\n                        let k_max_agenda_size = 100_000;\\n                        let k_min_agenda_size = 512;\\n                        if agenda.len() > k_max_agenda_size {\\n                            let mut new_agenda = BinaryHeap::new();\\n                            let len = min(k_min_agenda_size, n * 10);\\n                            for _i in 0..len {\\n                                new_agenda.push(agenda.pop().unwrap());\\n                            }\\n                            agenda = new_agenda;\\n                        }\\n                    }\\n                }\\n                hypotheses\\n            }\\n        }\\n    }\\n\\n    pub fn nbest_tokens(&mut self, n: usize) -> Vec<Vec<String>> {\\n        self.nbest(n)\\n            .iter()\\n            .map(|v| v.iter().map(|node| self.piece(&node.borrow())).collect())\\n            .collect()\\n    }\\n\\n    pub fn len(&self) -> usize {\\n        self.len\\n    }\\n\\n    pub fn is_empty(&self) -> bool {\\n        self.len == 0\\n    }\\n\\n    pub fn bos_node(&self) -> NodeRef {\\n        Rc::clone(&self.end_nodes[0][0])\\n    }\\n    pub fn eos_node(&self) -> NodeRef {\\n        Rc::clone(&self.begin_nodes[self.len][0])\\n    }\\n\\n    pub fn surface(&self, n: usize) -> &str {\\n        match self.sentence.char_indices().nth(n) {\\n            Some((pos, _)) => &self.sentence[pos..],\\n            None => \\"\\",\\n        }\\n    }\\n    pub fn sentence(&self) -> &str {\\n        self.sentence\\n    }\\n\\n    pub fn populate_marginal(&self, freq: f64, expected: &mut [f64]) -> f64 {\\n        let len = self.len();\\n        let n_nodes = self.nodes.len();\\n        let mut alpha = vec![0.0; n_nodes];\\n        let mut beta = vec![0.0; n_nodes];\\n        for pos in 0..=len {\\n            for rnode in &self.begin_nodes[pos] {\\n                for lnode in &self.end_nodes[pos] {\\n                    let lid = lnode.borrow().node_id;\\n                    let rid = rnode.borrow().node_id;\\n                    alpha[rid] = log_sum_exp(\\n                        alpha[rid],\\n                        lnode.borrow().score + alpha[lid],\\n                        *lnode == self.end_nodes[pos][0],\\n                    );\\n                }\\n            }\\n        }\\n        for pos in (0..=len).rev() {\\n            // let rpos = len - pos;\\n            for lnode in &self.end_nodes[pos] {\\n                for rnode in &self.begin_nodes[pos] {\\n                    let lid = lnode.borrow().node_id;\\n                    let rid = rnode.borrow().node_id;\\n                    beta[lid] = log_sum_exp(\\n                        beta[lid],\\n                        rnode.borrow().score + beta[rid],\\n                        *rnode == self.begin_nodes[pos][0],\\n                    );\\n                }\\n            }\\n        }\\n\\n        let eos_id = self.begin_nodes[len][0].borrow().node_id;\\n        let z = alpha[eos_id];\\n        for pos in 0..len {\\n            for node in &self.begin_nodes[pos] {\\n                let node_id = node.borrow().node_id;\\n                let id = node.borrow().id;\\n                let a = alpha[node_id];\\n                let b = beta[node_id];\\n                let total = a + node.borrow().score + b - z;\\n                let update = freq * total.exp();\\n                expected[id] += update;\\n            }\\n        }\\n        freq * z\\n    }\\n\\n    pub fn sample(&self, theta: f64) -> Vec<NodeRef> {\\n        let len = self.len();\\n        if len == 0 {\\n            return vec![];\\n        }\\n        let mut alpha = vec![0.0; self.nodes.len()];\\n        for pos in 0..=len {\\n            for rnode in &self.begin_nodes[pos] {\\n                for lnode in &self.end_nodes[pos] {\\n                    let lid = lnode.borrow().node_id;\\n                    let rid = rnode.borrow().node_id;\\n                    alpha[rid] = log_sum_exp(\\n                        alpha[rid],\\n                        theta * (lnode.borrow().score + alpha[lid]),\\n                        *lnode == self.end_nodes[pos][0],\\n                    );\\n                }\\n            }\\n        }\\n\\n        let mut rng = thread_rng();\\n        let mut results: Vec<NodeRef> = vec![];\\n        let mut probs: Vec<f64> = vec![];\\n        let mut z = alpha[self.eos_node().borrow().node_id];\\n        let mut node = self.eos_node();\\n        loop {\\n            probs.clear();\\n            let pos = node.borrow().pos;\\n            for lnode in &self.end_nodes[pos] {\\n                let lid = lnode.borrow().node_id;\\n                probs.push((alpha[lid] + theta * lnode.borrow().score - z).exp())\\n            }\\n            let dist = WeightedIndex::new(&probs).unwrap();\\n            let index = dist.sample(&mut rng);\\n            node = Rc::clone(&self.end_nodes[pos][index]);\\n            if node == self.bos_node() {\\n                break;\\n            }\\n            z = alpha[node.borrow().node_id];\\n            results.push(Rc::clone(&node));\\n        }\\n        results.reverse();\\n        results\\n    }\\n\\n    pub fn sample_token(&self, theta: f64) -> Vec<String> {\\n        self.sample(theta)\\n            .iter()\\n            .map(|node| self.piece(&node.borrow()))\\n            .collect()\\n    }\\n}\\n\\n#[cfg(test)]\\nmod tests {\\n    use super::*;\\n    use assert_approx_eq::assert_approx_eq;\\n\\n    #[test]\\n    fn set_sentence() {\\n        let lattice = Lattice::from(\\"\\", 1, 2);\\n\\n        assert_eq!(lattice.len(), 0);\\n\\n        let lattice = Lattice::from(\\"\\", 1, 2);\\n        assert_eq!(lattice.len(), 0);\\n        assert_eq!(lattice.sentence(), \\"\\");\\n        assert_eq!(lattice.surface(0), \\"\\");\\n\\n        let lattice = Lattice::from(\\"test\\", 1, 2);\\n        assert_eq!(lattice.len(), 4);\\n        assert_eq!(lattice.sentence(), \\"test\\");\\n        assert_eq!(lattice.surface(0), \\"test\\");\\n        assert_eq!(lattice.surface(1), \\"est\\");\\n        assert_eq!(lattice.surface(2), \\"st\\");\\n        assert_eq!(lattice.surface(3), \\"t\\");\\n\\n        let bos = lattice.bos_node();\\n        let eos = lattice.eos_node();\\n\\n        assert_eq!(bos.borrow().id, 1);\\n        assert_eq!(eos.borrow().id, 2);\\n        assert_eq!(\\n            lattice.end_nodes[0].first().unwrap().borrow().id,\\n            bos.borrow().id\\n        );\\n        assert_eq!(\\n            lattice.begin_nodes[4].first().unwrap().borrow().id,\\n            eos.borrow().id\\n        );\\n\\n        let lattice = Lattice::from(\\"\\u30c6\\u30b9\\u30c8ab\\", 1, 2);\\n        assert_eq!(lattice.len(), 11);\\n        assert_eq!(lattice.sentence(), \\"\\u30c6\\u30b9\\u30c8ab\\");\\n        assert_eq!(lattice.surface(0), \\"\\u30c6\\u30b9\\u30c8ab\\");\\n        assert_eq!(lattice.surface(1), \\"\\u30b9\\u30c8ab\\");\\n        assert_eq!(lattice.surface(2), \\"\\u30c8ab\\");\\n        assert_eq!(lattice.surface(3), \\"ab\\");\\n        assert_eq!(lattice.surface(4), \\"b\\");\\n    }\\n\\n    #[test]\\n    fn insert_test() {\\n        let mut lattice = Lattice::from(\\"AB\\u3042\\u3044\\", 1, 2);\\n\\n        lattice.insert(0, 1, 0.0, 3);\\n        lattice.insert(1, 1, 0.0, 4);\\n        lattice.insert(2, 3, 0.0, 5);\\n        lattice.insert(5, 3, 0.0, 6);\\n        lattice.insert(0, 2, 0.0, 7);\\n        lattice.insert(1, 4, 0.0, 8);\\n        lattice.insert(2, 6, 0.0, 9);\\n        // 0 & 1 are bos and eos\\n        let node0 = lattice.nodes[2].borrow();\\n        let node1 = lattice.nodes[3].borrow();\\n        let node2 = lattice.nodes[4].borrow();\\n        let node3 = lattice.nodes[5].borrow();\\n        let node4 = lattice.nodes[6].borrow();\\n        let node5 = lattice.nodes[7].borrow();\\n        let node6 = lattice.nodes[8].borrow();\\n\\n        assert_eq!(lattice.piece(&node0), \\"A\\");\\n        assert_eq!(lattice.piece(&node1), \\"B\\");\\n        assert_eq!(lattice.piece(&node2), \\"\\u3042\\");\\n        assert_eq!(lattice.piece(&node3), \\"\\u3044\\");\\n        assert_eq!(lattice.piece(&node4), \\"AB\\");\\n        assert_eq!(lattice.piece(&node5), \\"B\\u3042\\");\\n        assert_eq!(lattice.piece(&node6), \\"\\u3042\\u3044\\");\\n\\n        assert_eq!(node0.pos, 0);\\n        assert_eq!(node1.pos, 1);\\n        assert_eq!(node2.pos, 2);\\n        assert_eq!(node3.pos, 5);\\n        assert_eq!(node4.pos, 0);\\n        assert_eq!(node5.pos, 1);\\n        assert_eq!(node6.pos, 2);\\n\\n        assert_eq!(node0.length, 1);\\n        assert_eq!(node1.length, 1);\\n        assert_eq!(node2.length, 3);\\n        assert_eq!(node3.length, 3);\\n        assert_eq!(node4.length, 2);\\n        assert_eq!(node5.length, 4);\\n        assert_eq!(node6.length, 6);\\n\\n        assert_eq!(lattice.bos_node().borrow().id, 1);\\n        assert_eq!(lattice.eos_node().borrow().id, 2);\\n        assert_eq!(node0.id, 3);\\n        assert_eq!(node1.id, 4);\\n        assert_eq!(node2.id, 5);\\n        assert_eq!(node3.id, 6);\\n        assert_eq!(node4.id, 7);\\n        assert_eq!(node5.id, 8);\\n        assert_eq!(node6.id, 9);\\n\\n        assert_eq!(lattice.begin_nodes[0].len(), 2);\\n        assert_eq!(lattice.begin_nodes[1].len(), 2);\\n        assert_eq!(lattice.begin_nodes[2].len(), 2);\\n        assert_eq!(lattice.begin_nodes[5].len(), 1);\\n        assert_eq!(lattice.begin_nodes[8].len(), 1);\\n\\n        assert_eq!(lattice.end_nodes[0].len(), 1);\\n        assert_eq!(lattice.end_nodes[1].len(), 1);\\n        assert_eq!(lattice.end_nodes[2].len(), 2);\\n        assert_eq!(lattice.end_nodes[5].len(), 2);\\n        assert_eq!(lattice.end_nodes[8].len(), 2);\\n\\n        assert_eq!(lattice.begin_nodes[0][0].borrow().id, node0.id);\\n        assert_eq!(lattice.begin_nodes[0][1].borrow().id, node4.id);\\n        assert_eq!(lattice.begin_nodes[1][0].borrow().id, node1.id);\\n        assert_eq!(lattice.begin_nodes[1][1].borrow().id, node5.id);\\n        assert_eq!(lattice.begin_nodes[2][0].borrow().id, node2.id);\\n        assert_eq!(lattice.begin_nodes[2][1].borrow().id, node6.id);\\n        assert_eq!(lattice.begin_nodes[5][0].borrow().id, node3.id);\\n        assert_eq!(\\n            lattice.eos_node().borrow().id,\\n            lattice.begin_nodes[8][0].borrow().id\\n        );\\n\\n        assert_eq!(\\n            lattice.bos_node().borrow().id,\\n            lattice.end_nodes[0][0].borrow().id\\n        );\\n        assert_eq!(node0.id, lattice.end_nodes[1][0].borrow().id);\\n        assert_eq!(node1.id, lattice.end_nodes[2][0].borrow().id);\\n        assert_eq!(node4.id, lattice.end_nodes[2][1].borrow().id);\\n        assert_eq!(node2.id, lattice.end_nodes[5][0].borrow().id);\\n        assert_eq!(node5.id, lattice.end_nodes[5][1].borrow().id);\\n        assert_eq!(node3.id, lattice.end_nodes[8][0].borrow().id);\\n        assert_eq!(node6.id, lattice.end_nodes[8][1].borrow().id);\\n    }\\n\\n    #[test]\\n    fn test_viterbi() {\\n        let mut lattice = Lattice::from(\\"ABC\\", 1, 2);\\n        assert_eq!(lattice.viterbi(), vec![]);\\n        // Still incomplete\\n        lattice.insert(0, 1, 0.0, 3);\\n        assert_eq!(lattice.viterbi(), vec![]);\\n        lattice.insert(1, 1, 0.0, 4);\\n        lattice.insert(2, 1, 0.0, 5);\\n        // XXX: In sentence piece this is not tested, still incomplete ?\\n        assert_eq!(lattice.viterbi().len(), 3);\\n    }\\n\\n    #[test]\\n    fn test_viterbi2() {\\n        let mut lattice = Lattice::from(\\"ABC\\", 1, 2);\\n\\n        lattice.insert(0, 1, 0.0, 3);\\n        lattice.insert(1, 1, 0.0, 4);\\n        lattice.insert(2, 1, 0.0, 5);\\n\\n        assert_eq!(lattice.tokens(), [\\"A\\", \\"B\\", \\"C\\"]);\\n\\n        lattice.insert(0, 2, 2.0, 6);\\n        assert_eq!(lattice.tokens(), [\\"AB\\", \\"C\\"]);\\n\\n        lattice.insert(1, 2, 5.0, 7);\\n        assert_eq!(lattice.tokens(), [\\"A\\", \\"BC\\"]);\\n\\n        lattice.insert(0, 3, 10.0, 8);\\n        assert_eq!(lattice.tokens(), [\\"ABC\\"]);\\n    }\\n\\n    #[test]\\n    fn test_nbest() {\\n        let mut lattice = Lattice::from(\\"ABC\\", 1, 2);\\n        lattice.insert(0, 1, 0.0, 3);\\n        lattice.insert(1, 1, 0.0, 4);\\n        lattice.insert(2, 1, 0.0, 5);\\n        lattice.insert(0, 2, 2.0, 6);\\n        lattice.insert(1, 2, 5.0, 7);\\n        lattice.insert(0, 3, 10.0, 8);\\n\\n        let nbests = lattice.nbest_tokens(10);\\n        assert_eq!(\\n            nbests,\\n            vec![\\n                vec![\\"ABC\\"],\\n                vec![\\"A\\", \\"BC\\"],\\n                vec![\\"AB\\", \\"C\\"],\\n                vec![\\"A\\", \\"B\\", \\"C\\"]\\n            ]\\n        );\\n\\n        assert!(lattice.nbest_tokens(0).is_empty());\\n        assert_eq!(lattice.nbest_tokens(1), vec![vec![\\"ABC\\"]]);\\n    }\\n    #[test]\\n    fn test_log_sum_exp() {\\n        let mut x = 0.0;\\n\\n        let v: Vec<f64> = vec![1.0, 2.0, 3.0];\\n        for (i, y) in v.iter().enumerate() {\\n            x = log_sum_exp(x, *y, i == 0);\\n        }\\n        assert_approx_eq!(x, v.iter().map(|n| n.exp()).sum::<f64>().ln(), 0.001);\\n    }\\n\\n    #[test]\\n    fn test_populate() {\\n        let mut lattice = Lattice::from(\\"ABC\\", 1, 2);\\n        lattice.insert(0, 1, 1.0, 3); // A\\n        lattice.insert(1, 1, 1.2, 4); // B\\n        lattice.insert(2, 1, 2.5, 5); // C\\n        lattice.insert(0, 2, 3.0, 6); // AB\\n        lattice.insert(1, 2, 4.0, 7); // BC\\n        lattice.insert(0, 3, 2.0, 8); // ABC\\n\\n        let mut probs = vec![0.0; 9];\\n        let p1 = (1.0_f64 + 1.2 + 2.5).exp();\\n        let p2 = (3.0_f64 + 2.5).exp();\\n        let p3 = (1.0_f64 + 4.0).exp();\\n        let p4 = 2.0_f64.exp();\\n        let z = p1 + p2 + p3 + p4;\\n\\n        let log_z = lattice.populate_marginal(1.0, &mut probs);\\n\\n        assert_approx_eq!(log_z, z.ln(), 0.001);\\n        assert_approx_eq!(probs[0], 0.0, 0.001);\\n        assert_approx_eq!(probs[1], 0.0, 0.001);\\n        assert_approx_eq!(probs[2], 0.0, 0.001);\\n        assert_approx_eq!(probs[3], (p1 + p3) / z, 0.001);\\n        assert_approx_eq!(probs[4], (p1) / z, 0.001);\\n        assert_approx_eq!(probs[5], (p1 + p2) / z, 0.001);\\n        assert_approx_eq!(probs[6], (p2) / z, 0.001);\\n        assert_approx_eq!(probs[7], (p3) / z, 0.001);\\n        assert_approx_eq!(probs[8], (p4) / z, 0.001);\\n    }\\n}"}}}'

=============================


=============================

Sending:

b'Content-Length: 180\r\n\r\n{"jsonrpc": "2.0", "id": 2, "method": "textDocument/semanticTokens/full", "params": {"textDocument": {"uri": "file:///Users/ineilpaul/Downloads/Semantic_LSP/Rust/Tests/test2.rs"}}}'

=============================

Content-Length: 60

{"jsonrpc":"2.0","id":2,"result":{"resultId":"1","data":[]}}
=============================

Sending:

b'Content-Length: 160\r\n\r\n{"jsonrpc": "2.0", "method": "textDocument/didClose", "params": {"textDocument": {"uri": "file:///Users/ineilpaul/Downloads/Semantic_LSP/Rust/Tests/test2.rs"}}}'

=============================

Content-Length: 163

{"jsonrpc":"2.0","method":"textDocument/publishDiagnostics","params":{"uri":"file:///Users/ineilpaul/Downloads/Semantic_LSP/Rust/Tests/test2.rs","diagnostics":[]}}%  

I'm genuinely confused as to whether I am missing something here or I have stumbled upon a bug.

Start by running the Python reproduction script against the reported rust-analyzer and Rust versions, focusing on initialization, didOpen, and textDocument/semanticTokens/full. Compare the server's advertised semantic-token capability with its empty response for test2.rs; done means the request returns the expected tokens or the issue is explained by a reproducible protocol or setup problem.