Better syntax highlighting: switching from `highlight.js` to `shiki`

Problem

The current highlighting library highlight.js is lightweight and can run on both server and client, but provides a very basic syntax parsing that could be greatly improved.

Supersede #2313 #2292 #2268 #2238 #2237 #2360 #2445 #1652 ...

Proposed Solution

Consider switching to shiki, a high quality syntax highlighter that uses TextMate grammars and themes (same as VS Code), yet stays performant. It is also actively maintained.

There have been significant efforts and improvements that led to shiki-v1, and I believe that it's worthwile to make this migration and make shiki the default highlighter for mdbook. Today, mdbook is used to document a lot of code snippets and this change would definitely improve readability.

Example

Let's take Listing 20-24 from the rust book:

impl Worker {
    fn new(id: usize, receiver: Arc<Mutex<mpsc::Receiver<Job>>>) -> Worker {
        let thread = thread::spawn(move || loop {
            let message = receiver.lock().unwrap().recv();

            match message {
                Ok(job) => {
                    println!("Worker {id} got a job; executing.");

                    job();
                }
                Err(_) => {
                    println!("Worker {id} disconnected; shutting down.");
                    break;
                }
            }
        });

        Worker {
            id,
            thread: Some(thread),
        }
    }
}

mdbook w/ highlight.js (Coal theme)	Shiki (github-dark-dimmed theme)
Open in browser	Open in browser

Vs-code reference (theme Github Dark Dimmed):

Implementation example

@cestef overriding of theme/book.js for shiki compatibility as explained here

Notes

Shiki v1: https://nuxt.com/blog/shiki-v1
Another alternative: https://github.com/wooorm/starry-night

Further ideas

Twoslash allows to extend Typescript code snippets from markdown file with extended Typescript compiler's information and vs code syntax engine. It is maintained and integrated perfectly with shiki.

With the strong rust-analyzer and the recent abstraction of the twoslash API in twoslash-protocol, it would make sense to build a twoslash-rs for mdbook. Some preliminary explorations have been done: by ayazhafiz/twoslash-rust and @jxom

This first implementation of Shiki in mdbook would pave the way for this later on.

Start by reviewing mdBook's current highlight.js integration and the linked docs/theme/book.js example for Shiki compatibility. Define the migration's handling of themes, languages, client/server behavior, and existing code blocks; done means Shiki is the default highlighter without breaking current documentation output.