Move away from `pickle`-based serialisation

Is your feature request related to a problem? Please describe.

Currently, Sphinx uses Python's pickle module for various serialisation-related functions:

• Saving and restoring the build environment (app.env) for incremental rebuilds
• Saving and restoring the Docutils AST ('doctree') of individual documents between the reading and writing phases
• Reconcilling changes in the build environment between worker processes and the main process, when using parallel processing
• Saving coverage data in sphinx.ext.coverage

The pickle module is convenient, but also insecure (unpickling can execute arbitrary Python code) and hard to introspect. I would like to replace our use of pickle with data-only serialisation.

Challenges include, but are not limited to:

• There is no round-trip serialisation format for Docutils ASTs.
• Extensions currently store arbitrary data directly on app.env, which includes arbitrary (non-builtin) types.

For non-pickle serialisation to be supported by extensions, I expect we will need to provide (a) hooks for serialising and deserialising and (b) an 'official' method of saving data to the environment.

In terms of implementation, we progressively convert attributes or data to be (de)serialised via a non-pickle mechanism.

Describe the solution you'd like

To eventually deprecate and remove all use of the pickle module.

Describe alternatives you've considered

The status quo.

A

cc @sphinx-doc/developers @sphinx-doc/triagers for input.

Start by auditing Sphinx's pickle use for app.env, individual document doctrees, parallel-processing reconciliation, and sphinx.ext.coverage. Done means these paths progressively use data-only serialisation, with hooks for extensions and an official way to save environment data, eventually allowing pickle to be deprecated and removed.