Improve CodeBlock Extensibility

### Have you read the Contributing Guidelines on issues?

- [x] I have read the [Contributing Guidelines on issues](https://github.com/facebook/docusaurus/blob/main/CONTRIBUTING.md#issues).

### Description

The main goal of this feature is to allow developers to extend the syntax highlighting in a react/docusaurus compatible way and by this work around the limitations of Prismjs plugins not being available.

### Has this been requested on Canny?

_No response_

### Motivation

Docusuarus / prism-react-renderer do not allow using standard prismjs plugins due to the DOM manipulation nature of them. It has been discussed that Prism plugins are not supported, and it also was discussed to use alternative components for the syntax highlighting needs.

* https://github.com/facebook/docusaurus/issues/3318
* https://github.com/facebook/docusaurus/issues/9122

With this proposal developers would get a mechanism of enriching the syntax highlighting by utilizing magic comments and swizzling without major impact to the Docusaurus core functionality.

My personal goal is to develop a plugin where I can generate hyperlinks for individual tokens rendered to let user jump to the related reference documentation. The current state of implementation requires me to swizzle and adapt a major part of the CodeBlock components to achieve this.

### API design

As of today Docusaurus offers:
1. Magic Comments as a means of highlighting individual lines via CSS classes.
2. Metadata string as a means of injecting simple options
3. Fine grained swizzling as mechanism to allow customizing the component rendering.

Based on this strategy also the extensibility of CodeBlocks should be designed. In a first step, the goal is to allow individual website authors to customize the components to their needs. It is at this point a non-goal to provide a fully fledged plugin system where you could pull NPM packages and register them. This could become an option in future.

The concrete proposal is to:

1. Provide a mechanism for end users to inject "plugin" configurations. (optional but recommended)
1.1. Parse Metadata string into a option bag.

The metadata string is currently a plain string rather parsed individually at some spots. Docusaurus should attempt parsing the options with a defined syntax:

```antlr
grammar ExprParser;

metadataComment

metadata
: rangeSyntax? optionSyntax* EOF;

rangeSyntax: '{' lineRange (',' lineRange)* '}';

lineRange: INT
| INT '-' INT;

optionSyntax: ID
| ID '=' ID
| ID '=' STRING;

INT : [0-9]+ ;
ID: [a-zA-Z_][a-zA-Z_0-9]* ;
WS: [ \t\n\r\f]+ -> skip ;
STRING : '"' ~[<"]* '"' | '\'' ~[<']* '\'';
```


The options are then parsed into a defined data structure for easy access in later steps. Also docusaurus could internally benefit from this bag instead of doing "contains" checks.

https://github.com/facebook/docusaurus/blob/3782244ce7fd95fdb3f94444884ae5a1a85da915/packages/docusaurus-theme-common/src/utils/codeBlockUtils.ts#L155-L165

options are meant to be unique and are overwritten. Support for arrays or maps as values could be added later if the need arises.

1.2. Metadata Magic Comments
The metadata string requires all options to be placed in a single line which has quite some limits when you need to specify many options. Hence it would be a good extension to also allow specifying the options from 1.1 via some magic comment syntax.

These magic comments would be parsed into the property bag and then erased.

For a simple initial version we parse the whole code for magic comments and fill the bag at once. Later this could be extended that options can be changed at any time in the code affecting subsequent lines.

An option comment starts with a configurable special prefix (to be defined), should not interfere with the majority of real comments, and only allows one option per comment. The configuration option `prism: { optionCommentPrefix: '!docusuarus-' }` allows users to eliminate conflicts.

```
// !docusaurus-option
// !docusuaurs-option2=true
// !docusaurus-option3=value
// !docusaurus-option4="long value"
```

2. Create a new swizzlable component for rendering individual tokens and ensure they receive the metadata bag. (required for MVP)
2.1 Add a swizzleable `LineToken`

Currently the `Line` component creates a `span` preventing any customization of the rendered token.

https://github.com/facebook/docusaurus/blob/3782244ce7fd95fdb3f94444884ae5a1a85da915/packages/docusaurus-theme-classic/src/theme/CodeBlock/Line/index.tsx#L44

A new component allows swizzling and customizing the rendering on token level (e.g. a dev could create a `` instead).

2.2. Pass through the metadata options.

The parsed options from 1. should be passed down to the `Line` and `LineToken` components so devs can make use of it. If we decide not to parse the metadata string (yet?) at least the plain string available on `String.tsx` should be passed through.

### Have you tried building it?

I successfully created a rough prototype locally by swizzling `CodeBlock` and `Line` to pass-through the metadata and customize the generated token. But it requires swizzling of various internal components which is risky.


### Self-service

- [x] I'd be willing to contribute this feature to Docusaurus myself.