Certificates for signed exchanges should be allowed also for TLS connections.

I read this in the specification:
"Clients MUST NOT accept certificates with this extension in TLS connections (Section 4.4.2.2 of [RFC8446])."

I don't see any specific reason for this. I would suggest removing this completely from specification, and allowing content to be signed with the same certificate as for TLS connections. (provided the certificate is allowed for signed exchanges)

Since a signed exchange have a very specific format, I don't see any risk with for example an site-wide XSS becoming a signing oracle for signing exchanges, unless the administrator explicitly configures the page/server in that way for some specific application. (for example a timestamping service allowing signing of arbitary data which is then time-stamped)

Another way to improve on this, is to have a special magic value for the extension, that will allow dual usage, so if the extension, instead of having the null value ( 05 00 ), have the exact octet-binary value "BOTH" ( 04 42 4f 54 48 ), it will allow dual usage of the certificate.

Thus specification could be instead:
"A conforming CA MUST NOT issue certificates with this extension unless, for each dNSName in the subjectAltName extension of the certificate to be issued:
An “issue” or “issuewild” CAA property ([RFC6844]) exists that authorizes the CA to issue the certificate; and
The “cansignhttpexchanges” parameter (Section 4.2.1) is present on the property and is equal to “yes” or "both".
A conforming CA MUST NOT issue certificates with this extension set to "BOTH", unless "cansignhttpexchanges" is equal to "both".

Clients MUST NOT accept certificates with this extension in TLS connections, unless the extension has the exact octet-binary value BOTH ( 04 42 4f 54 48 ) (Section 4.4.2.2 of [RFC8446])."

For this, the cansignhttpexchanges CAA value must also be extended with a "both" option to allow certificates to be issued with the BOTH extension value. (if "yes", it will only allow NULL, but if "both", it will alllow both NULL and BOTH values)

I also read this:
"TLS server certificates must be accessible from online servers, so they’re easier to steal or use as signing oracles than an offline key. An exchange’s signing key doesn’t need to be online."
and this:
"Using an exchange-signing key in a TLS (or other directly-internet-facing) server increases the risk that an attacker can steal the private key, which will allow them to mint packages (similar to Section 6.4) until their theft is discovered."

This is very untrue, as most content-signing servers will of course be directly internet-facing. A majority of server operators will just configure their servers to sign static content when requested, or possibly with a client whitelist (like cloudflares IPs) to prevent it from signing content for others.
Thus their CanSignHttpExchanges certificate will in most cases, reside side-with-side with their TLS certificate in the same folder on the same machine.

Only more sensitive and advanced companies will make content-signing servers airgapped.

The only way to enforce secure usage of such a certificate and make sure its "airgapped", is to require it to be loaded on a secure element with presence validation. This is a requirement imposed by most CA's for sub-CA certificates.

The only risk I see with completely dropping the restriction, is that a stolen content-signing certificate could be used to host a TLS server used for example phishing attacks and such.

Thats why it might be good to instead use the proposed NULL and BOTH values, so theres 3 types of certificates:
1: Certificates ONLY valid for TLS connections.
2: Certificates valid for BOTH.
3: Certificates ONLY valid for Signed-exchanges.

thus a server operator is able to choose the security level that fits them best.

Start by reading the cited specification sections 4.2.1 and 4.4.2.2 of RFC8446, then review the 19-comment thread for an unresolved decision. The proposed work concerns allowing a BOTH certificate value for TLS and signed exchanges, with corresponding CAA behavior; done would require agreement on the security model and an accepted specification change.