Any runtime exception in AbstractBaseDispatcher breaks the broker

**Describe the bug**
I am playing with KOP and I was able to see the error below.
The point for this issue is not the problem itself but the fact that after that error the broker stops processing requests, letting every client hang forever (until timeouts...).

We should have some handling of RuntimeExceptions in a way that the broker can recover as much as possible.
```

java.lang.IllegalArgumentException: Invalid unknonwn tag type: 6 │
│ pulsar-broker at org.apache.pulsar.common.api.proto.LightProtoCodec.skipUnknownField(LightProtoCodec.java:270) ~[com.datastax.oss-pulsar-common-2.8.0.1.1.6-SNAPSHOT.jar:2.8.0.1.1.6-SNAPSHOT] │
│ pulsar-broker at org.apache.pulsar.common.api.proto.MessageMetadata.parseFrom(MessageMetadata.java:1370) ~[com.datastax.oss-pulsar-common-2.8.0.1.1.6-SNAPSHOT.jar:2.8.0.1.1.6-SNAPSHOT] │
│ pulsar-broker at org.apache.pulsar.common.protocol.Commands.parseMessageMetadata(Commands.java:429) ~[com.datastax.oss-pulsar-common-2.8.0.1.1.6-SNAPSHOT.jar:2.8.0.1.1.6-SNAPSHOT] │
│ pulsar-broker at org.apache.pulsar.common.protocol.Commands.parseMessageMetadata(Commands.java:416) ~[com.datastax.oss-pulsar-common-2.8.0.1.1.6-SNAPSHOT.jar:2.8.0.1.1.6-SNAPSHOT] │
│ pulsar-broker at org.apache.pulsar.common.protocol.Commands.peekMessageMetadata(Commands.java:1673) ~[com.datastax.oss-pulsar-common-2.8.0.1.1.6-SNAPSHOT.jar:2.8.0.1.1.6-SNAPSHOT] │
│ pulsar-broker at org.apache.pulsar.broker.service.AbstractBaseDispatcher.filterEntriesForConsumer(AbstractBaseDispatcher.java:122) ~[com.datastax.oss-pulsar-broker-2.8.0.1.1.6-SNAPSHOT.jar:2.8.0.1.1.6-SNA │
│ pulsar-broker at org.apache.pulsar.broker.service.AbstractBaseDispatcher.filterEntriesForConsumer(AbstractBaseDispatcher.java:101) ~[com.datastax.oss-pulsar-broker-2.8.0.1.1.6-SNAPSHOT.jar:2.8.0.1.1.6-SNA │
│ pulsar-broker at org.apache.pulsar.broker.service.persistent.PersistentDispatcherSingleActiveConsumer.internalReadEntriesComplete(PersistentDispatcherSingleActiveConsumer.java:202) ~[com.datastax.oss-puls │
│ pulsar-broker at org.apache.pulsar.broker.service.persistent.PersistentDispatcherSingleActiveConsumer.lambda$readEntriesComplete$1(PersistentDispatcherSingleActiveConsumer.java:146) ~[com.datastax.oss-pul │
│ pulsar-broker at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [com.datastax.oss-managed-ledger-2.8.0.1.1.6-SNAPSHOT.jar:2.8.0.1.1.6-SNAPSHOT] │
│ pulsar-broker at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.14.2.jar:4.14.2] │
│ pulsar-broker at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] │
│ pulsar-broker at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] │
│ pulsar-broker at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?] │
│ pulsar-broker at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] │
│ pulsar-broker at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] │
│ pulsar-broker at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.68.Final.jar:4.1.68.Final] │
│ pulsar-broker at java.lang.Thread.run(Thread.java:829) [?:?]
```

**Additional context**
I am on Pulsar 2.8.x (actually DataStax fork, but the code is the same regarding this issue)

Start at broker service AbstractBaseDispatcher.java, especially filterEntriesForConsumer, and trace the call path shown through PersistentDispatcherSingleActiveConsumer.internalReadEntriesComplete. Reproduce the RuntimeException scenario from the report and determine how the broker should recover; done means subsequent client requests continue processing instead of hanging until timeout.

Written by the indexing model from the issue text.