llvm / llvm/llvm-project

[clangd] assertion` It != Files.end()` in clang::syntax::TokenBuffer::spelledTokens(FileID) on `codeAction`

Open
#199,608 1 comment 1 reaction 0 assignees View on GitHub
clangd crash generated by fuzzer
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

This crash requires a small workspace repro rather than a standalone single file.

More precisely, it involves three files under:
https://github.com/llvm/llvm-project/tree/main/clang/test/Lexer/
Main file, reduced from https://github.com/llvm/llvm-project/blob/main/clang/test/Lexer/SourceLocationsOverflow.c SourceLocationsOverflow.c
```
#include "Inputs/inc1.h"
#include "Inputs/inc1.h"
#include "Inputs/inc1.h"
#include "Inputs/inc1.h"
#include "Inputs/inc1.h"
#include "Inputs/inc1.h"
#include "Inputs/inc1.h"
#include "Inputs/inc1.h"
#include "Inputs/inc1.h"
#include "Inputs/inc1.h"
#include "Inputs/inc1.h"
#include "Inputs/inc1.h"
#include "Inputs/inc1.h"
#include "Inputs/inc1.h"
#include "Inputs/inc1.h"
```

The following additional files are also required:, https://github.com/llvm/llvm-project/blob/main/clang/test/Lexer/Inputs/inc1.h and https://github.com/llvm/llvm-project/blob/main/clang/test/Lexer/Inputs/inc2.h
The minimal workspace layout is
```
SourceLocationsOverflow.c
-> Inputs/inc1.h
-> inc2.h
```

compile command:
```
compile command:
clang -std=c2y -fsyntax-only ...
```
In clangd, this command is provided through `compile_commands.json`.

requests:
```
initialize -> initialized -> didOpen -> textDocument/codeAction-> shutdown -> exit
```
log
```
I[12:07:58.197] clangd version 23.0.0git (https://github.com/llvm/llvm-project.git c75e1e62d11be965a4dca0d816a4d19a6684b043)
I[12:07:58.197] Features: linux+debug-tidy
I[12:07:58.197] PID: 20579
I[12:07:58.197] Working directory: /tmp/case035-reduce.EnD8tG/workspace
I[12:07:58.197] argv[0]: /tmp/build-assert-clangd-main-20260506/bin/clangd
I[12:07:58.198] Starting LSP over stdin/stdout
I[12:07:58.198] <-- initialize(1)
I[12:07:58.200] --> reply:initialize(1) 2 ms
I[12:07:58.375] <-- initialized
I[12:07:58.375] <-- textDocument/didOpen
I[12:07:58.378] Loaded compilation database from /tmp/case035-reduce.EnD8tG/workspace/compile_commands.json
[/harness/cases/staging/CASE-20260525-035/workspace]
/usr/lib/llvm-18/bin/clang -fsyntax-only -resource-dir=/tmp/build-assert-clangd-main-20260506/lib/clang/23 -std=c2y -- /tmp/case035-reduce.EnD8tG/workspace/clang/test/Lexer/SourceLocationsOverflow.c
I[12:07:58.378] Loaded compilation database from /harness/cases/staging/CASE-20260525-035/workspace/compile_commands.json
E[12:07:58.513] Could not build a preamble for file /tmp/case035-reduce.EnD8tG/workspace/clang/test/Lexer/SourceLocationsOverflow.c version 1: Could not emit PCH
E[12:07:58.513] error: in included file: translation unit is too large for Clang to process: ran out of source locations
I[12:07:58.636] --> textDocument/publishDiagnostics
I[12:07:58.725] <-- textDocument/codeAction(2)
clangd: /tmp/llvm-project-main-20260504/clang/lib/Tooling/Syntax/Tokens.cpp:380: llvm::ArrayRef clang::syntax::TokenBuffer::spelledTokens(clang::FileID) const: Assertion `It != Files.end()' failed.
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace and instructions to reproduce the bug.
I[12:07:58.975] <-- shutdown(3)
I[12:07:58.975] --> reply:shutdown(3) 0 ms
I[12:07:59.075] <-- exit
I[12:07:59.075] LSP finished, exiting with status 0
```

backtrace
```
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace and instructions to reproduce the bug.
I[02:17:37.372] <-- shutdown(2)
I[02:17:37.372] --> reply:shutdown(2) 0 ms
I[02:17:37.432] <-- exit
I[02:17:37.432] LSP finished, exiting with status 0
#0 0x00005e85edbc38f2 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /tmp/llvm-project-main-20260504/llvm/lib/Support/Unix/Signals.inc:884:3
#1 0x00005e85edbc066c llvm::sys::RunSignalHandlers() /tmp/llvm-project-main-20260504/llvm/lib/Support/Signals.cpp:108:20
#2 0x00005e85edbc0ee1 SignalHandler(int, siginfo_t*, void*) /tmp/llvm-project-main-20260504/llvm/lib/Support/Unix/Signals.inc:448:14
#3 0x000071f112a45330 (/lib/x86_64-linux-gnu/libc.so.6+0x45330)
#4 0x000071f112a9eb2c __pthread_kill_implementation ./nptl/pthread_kill.c:44:76
#5 0x000071f112a9eb2c __pthread_kill_internal ./nptl/pthread_kill.c:78:10
#6 0x000071f112a9eb2c pthread_kill ./nptl/pthread_kill.c:89:10
#7 0x000071f112a4527e raise ./signal/../sysdeps/posix/raise.c:27:6
#8 0x000071f112a288ff abort ./stdlib/abort.c:81:7
#9 0x000071f112a2881b _nl_load_domain ./intl/loadmsgcat.c:1177:9
#10 0x000071f112a3b517 (/lib/x86_64-linux-gnu/libc.so.6+0x3b517)
#11 0x00005e85eef8a469 clang::syntax::TokenBuffer::spelledTokens(clang::FileID) const /tmp/llvm-project-main-20260504/clang/lib/Tooling/Syntax/Tokens.cpp:380:3
#12 0x00005e85ef22ef79 std::iterator_traits::difference_type std::__distance(clang::syntax::Token const*, clang::syntax::Token const*, std::random_access_iterator_tag) /usr/include/c++/13/bits/stl_iterator_base_funcs.h:106:23
#13 0x00005e85ef22ef79 std::iterator_traits::difference_type std::distance(clang::syntax::Token const*, clang::syntax::Token const*) /usr/include/c++/13/bits/stl_iterator_base_funcs.h:151:29
#14 0x00005e85ef22ef79 partition_point > /usr/include/c++/13/bits/stl_algo.h:572:42
#15 0x00005e85ef22ef79 partition_point&, clang::clangd::(anonymous namespace)::SelectionTester::SelectionTester(const clang::syntax::TokenBuffer&, clang::FileID, unsigned int, unsigned int, const clang::SourceManager&):: > /tmp/llvm-project-main-20260504/llvm/include/llvm/ADT/STLExtras.h:2129:30
#16 0x00005e85ef22ef79 SelectionTester /tmp/llvm-project-main-20260504/clang-tools-extra/clangd/Selection.cpp:259:30
#17 0x00005e85ef22ef79 SelectionVisitor /tmp/llvm-project-main-20260504/clang-tools-extra/clangd/Selection.cpp:755:27
#18 0x00005e85ef22ef79 collect /tmp/llvm-project-main-20260504/clang-tools-extra/clangd/Selection.cpp:618:57
#19 0x00005e85ef22ef79 clang::clangd::SelectionTree::SelectionTree(clang::ASTContext&, clang::syntax::TokenBuffer const&, unsigned int, unsigned int) /tmp/llvm-project-main-20260504/clang-tools-extra/clangd/Selection.cpp:1127:36
#20 0x00005e85ef230aa4 std::_Deque_iterator::_Deque_iterator() /usr/include/c++/13/bits/stl_deque.h:152:9
#21 0x00005e85ef230aa4 std::_Deque_base>::_Deque_impl_data::_Deque_impl_data() /usr/include/c++/13/bits/stl_deque.h:517:29
#22 0x00005e85ef230aa4 std::_Deque_base>::_Deque_impl::_Deque_impl(std::allocator&&) /usr/include/c++/13/bits/stl_deque.h:558:33
#23 0x00005e85ef230aa4 std::_Deque_base>::_Deque_base(std::_Deque_base>&&) /usr/include/c++/13/bits/stl_deque.h:476:9
#24 0x00005e85ef230aa4 std::deque>::deque(std::deque>&&) /usr/include/c++/13/bits/stl_deque.h:933:7
#25 0x00005e85ef230aa4 clang::clangd::SelectionTree::SelectionTree(clang::clangd::SelectionTree&&) /tmp/llvm-project-main-20260504/clang-tools-extra/clangd/Selection.h:103:3
#26 0x00005e85ef230aa4 llvm::function_ref::operator()(clang::clangd::SelectionTree) const /tmp/llvm-project-main-20260504/llvm/include/llvm/ADT/STLFunctionalExtras.h:69:12
#27 0x00005e85ef230aa4 clang::clangd::SelectionTree::createEach(clang::ASTContext&, clang::syntax::TokenBuffer const&, unsigned int, unsigned int, llvm::function_ref) /tmp/llvm-project-main-20260504/clang-tools-extra/clangd/Selection.cpp:1096:16
#28 0x00005e85ef069ea4 __gnu_cxx::__normal_iterator> const*, std::vector>, std::allocator>>>>::__normal_iterator(std::unique_ptr> const* const&) /usr/include/c++/13/bits/stl_iterator.h:1077:9
#29 0x00005e85ef069ea4 std::vector>, std::allocator>>>::begin() const /usr/include/c++/13/bits/stl_vector.h:884:16
#30 0x00005e85ef069ea4 std::vector>, std::allocator>>>::empty() const /usr/include/c++/13/bits/stl_vector.h:1089:21
#31 0x00005e85ef069ea4 clang::clangd::(anonymous namespace)::tweakSelection(clang::clangd::Range const&, clang::clangd::InputsAndAST const&, llvm::vfs::FileSystem*) /tmp/llvm-project-main-20260504/clang-tools-extra/clangd/ClangdServer.cpp:671:3
#32 0x00005e85ef0750f9 llvm::Expected>, std::allocator>>>>::operator bool() /tmp/llvm-project-main-20260504/llvm/include/llvm/Support/Error.h:576:17
#33 0x00005e85ef0750f9 clang::clangd::ClangdServer::codeAction(clang::clangd::ClangdServer::CodeActionInputs const&, llvm::unique_function)>)::'lambda'(llvm::Expected)::operator()(llvm::Expected) /tmp/llvm-project-main-20260504/clang-tools-extra/clangd/ClangdServer.cpp:736:10
#34 0x00005e85ef076040 llvm::Expected::assertIsChecked() const /tmp/llvm-project-main-20260504/llvm/include/llvm/Support/Error.h:729:9
#35 0x00005e85ef076040 llvm::Expected::~Expected() /tmp/llvm-project-main-20260504/llvm/include/llvm/Support/Error.h:566:20
#36 0x00005e85ef076040 void llvm::detail::UniqueFunctionBase>::CallImpl)>)::'lambda'(llvm::Expected)>(void*, llvm::Expected&) /tmp/llvm-project-main-20260504/llvm/include/llvm/ADT/FunctionExtras.h:212:16
#37 0x00005e85ef2ae4f3 llvm::Expected::assertIsChecked() const /tmp/llvm-project-main-20260504/llvm/include/llvm/Support/Error.h:729:9
#38 0x00005e85ef2ae4f3 llvm::Expected::~Expected() /tmp/llvm-project-main-20260504/llvm/include/llvm/Support/Error.h:566:20
#39 0x00005e85ef2ae4f3 clang::clangd::(anonymous namespace)::ASTWorker::runWithAST(llvm::StringRef, llvm::unique_function)>, clang::clangd::TUScheduler::ASTActionInvalidation)::'lambda'()::operator()() /tmp/llvm-project-main-20260504/clang-tools-extra/clangd/TUScheduler.cpp:1012:11
#40 0x00005e85ef2a72aa clang::clangd::(anonymous namespace)::ASTWorker::runTask(llvm::StringRef, llvm::function_ref) /tmp/llvm-project-main-20260504/clang-tools-extra/clangd/TUScheduler.cpp:1328:1
#41 0x00005e85ef2aa9fa clang::clangd::(anonymous namespace)::ASTWorker::run() /tmp/llvm-project-main-20260504/clang-tools-extra/clangd/TUScheduler.cpp:1462:5
#42 0x00005e85ef48db6e llvm::detail::UniqueFunctionBase::operator=(llvm::detail::UniqueFunctionBase&&) /tmp/llvm-project-main-20260504/llvm/include/llvm/ADT/FunctionExtras.h:329:5
#43 0x00005e85ef48db6e llvm::detail::UniqueFunctionBase::operator=(llvm::detail::UniqueFunctionBase&&) /tmp/llvm-project-main-20260504/llvm/include/llvm/ADT/FunctionExtras.h:321:23
#44 0x00005e85ef48db6e llvm::unique_function::operator=(llvm::unique_function&&) /tmp/llvm-project-main-20260504/llvm/include/llvm/ADT/FunctionExtras.h:352:20
#45 0x00005e85ef48db6e operator() /tmp/llvm-project-main-20260504/clang-tools-extra/clangd/support/Threading.cpp:103:14
#46 0x00005e85ef48db6e operator())::&> /tmp/llvm-project-main-20260504/llvm/include/llvm/Support/thread.h:46:16
#47 0x00005e85ef48db6e __invoke_impl):: > >(void*)::, clang::clangd::AsyncTaskRunner::runAsync(const llvm::Twine&, llvm::unique_function)::&> /usr/include/c++/13/bits/invoke.h:61:36
#48 0x00005e85ef48db6e __invoke):: > >(void*)::, clang::clangd::AsyncTaskRunner::runAsync(const llvm::Twine&, llvm::unique_function)::&> /usr/include/c++/13/bits/invoke.h:96:40
#49 0x00005e85ef48db6e __apply_impl):: > >(void*)::, std::tuple):: >&, 0> /usr/include/c++/13/tuple:2302:27
#50 0x00005e85ef48db6e apply):: > >(void*)::, std::tuple):: >&> /usr/include/c++/13/tuple:2313:31
#51 0x00005e85ef48db6e GenericThreadProxy):: > > /tmp/llvm-project-main-20260504/llvm/include/llvm/Support/thread.h:44:15
#52 0x00005e85ef48db6e void* llvm::thread::ThreadProxy)::'lambda0'()>>(void*) /tmp/llvm-project-main-20260504/llvm/include/llvm/Support/thread.h:62:36
#53 0x000071f112a9caa4 start_thread ./nptl/pthread_create.c:447:8
#54 0x000071f112b29c6c clone3 ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:80:0
Signalled during AST worker action: codeAction
Filename: clang/test/Lexer/SourceLocationsOverflow.c
Directory: /tmp/.tmpnq8KZM
Command Line: /usr/lib/llvm-18/bin/clang -std=c2y -fsyntax-only -resource-dir=/tmp/build-assert-clangd-main-20260506/lib/clang/23 -- /tmp/.tmpnq8KZM/clang/test/Lexer/SourceLocationsOverflow.c
Version: 1
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.