issues decoding deneb payloads

Describe the bug
we're not entirely sure of the cause, only of the obvious error / panic

{
  "err": "BUG: count '130' higher than limit '8'",
  "level": "error",
  "method": "POST",
  "msg": "http request panic: POST /eth/v1/builder/blinded_blocks",
  "service": "relay/api",
  "time": "2025-05-12T21:00:25.321447032Z",
  "trace": "goroutine 1101784 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:26 +0x5e\ngithub.com/flashbots/mev-boost-relay/services/api.(*RelayAPI).getRouter.LoggingMiddlewareLogrus.func2.1()\n\tgithub.com/flashbots/go-utils@v0.11.0/httplogger/httplogger.go:132 +0x19e\npanic({0xfc1620?, 0xc01b844f20?})\n\truntime/panic.go:792 +0x132\ngithub.com/ferranbt/fastssz.(*Hasher).merkleizeImpl(0x1040?, {0xc01f0b7420?, 0x47dade?, 0x1300?}, {0xc01f0b7420?, 0xc01b831100?, 0x100?}, 0x8)\n\tgithub.com/ferranbt/fastssz@v0.1.4/hasher.go:364 +0x59a\ngithub.com/ferranbt/fastssz.(*Hasher).MerkleizeWithMixin(0xc000272680, 0x120, 0x81fa, 0xc01f0ad300?)\n\tgithub.com/ferranbt/fastssz@v0.1.4/hasher.go:288 +0x177\ngithub.com/ferranbt/fastssz.(*Hasher).PutBitlist(0xc000272680, {0xc01f0ad300?, 0x100?, 0x120?}, 0x800)\n\tgithub.com/ferranbt/fastssz@v0.1.4/hasher.go:232 +0x219\ngithub.com/attestantio/go-eth2-client/spec/phase0.(*Attestation).HashTreeRootWith(0xc0002e6200, {0x135b2f0, 0xc000272680})\n\tgithub.com/attestantio/go-eth2-client@v0.25.0/spec/phase0/attestation_ssz.go:114 +0x63\ngithub.com/attestantio/go-eth2-client/api/v1/deneb.(*BlindedBeaconBlockBody).HashTreeRootWith(0xc01b824780, {0x135b2f0, 0xc000272680})\n\tgithub.com/attestantio/go-eth2-client@v0.25.0/api/v1/deneb/blindedbeaconblockbody_ssz.go:507 +0x369\ngithub.com/attestantio/go-eth2-client/api/v1/deneb.(*BlindedBeaconBlock).HashTreeRootWith(0xc01c5a67e0, {0x135b2f0, 0xc000272680})\n\tgithub.com/attestantio/go-eth2-client@v0.25.0/api/v1/deneb/blindedbeaconblock_ssz.go:128 +0xc6\ngithub.com/ferranbt/fastssz.HashWithDefaultHasher({0x134a028, 0xc01c5a67e0})\n\tgithub.com/ferranbt/fastssz@v0.1.4/hasher.go:47 +0x73\ngithub.com/attestantio/go-eth2-client/api/v1/deneb.(*BlindedBeaconBlock).HashTreeRoot(...)\n\tgithub.com/attestantio/go-eth2-client@v0.25.0/api/v1/deneb/blindedbeaconblock_ssz.go:108\ngithub.com/attestantio/go-eth2-client/api.(*VersionedSignedBlindedBeaconBlock).Root(0xebf4000bc40c0002?)\n\tgithub.com/attestantio/go-eth2-client@v0.25.0/api/versionedsignedblindedbeaconblock.go:166 +0x199\ngithub.com/flashbots/mev-boost-relay/services/api.verifyBlockSignature(0xc038bc3290, {0x0, 0x0, 0x0, 0x0, 0x6a, 0x95, 0xa1, 0xa9, 0x67, ...}, ...)\n\tgithub.com/flashbots/mev-boost-relay/services/api/utils.go:169 +0x45\ngithub.com/flashbots/mev-boost-relay/services/api.(*RelayAPI).checkProposerSignature(0x286d01c2cb71a2ad?, 0xc725ea1f12e42f32?, {0xc01f09029c?, 0x9299d0933eb1fb9f?, 0x70a4a9402cc7d2e9?})\n\tgithub.com/flashbots/mev-boost-relay/services/api/service.go:1468 +0x113\ngithub.com/flashbots/mev-boost-relay/services/api.(*RelayAPI).handleGetPayload(0xc000244008, {0x1348858, 0xc01b60bf20}, 0xc01b824500)\n\tgithub.com/flashbots/mev-boost-relay/services/api/service.go:1774 +0x27b5\nnet/http.HandlerFunc.ServeHTTP(0xc01b8243c0?, {0x1348858?, 0xc01b60bf20?}, 0x2?)\n\tnet/http/server.go:2294 +0x29\ngithub.com/gorilla/mux.(*Router).ServeHTTP(0xc000052300, {0x1348858, 0xc01b60bf20}, 0xc01b824140)\n\tgithub.com/gorilla/mux@v1.8.1/mux.go:212 +0x1e2\ngithub.com/flashbots/mev-boost-relay/services/api.(*RelayAPI).getRouter.LoggingMiddlewareLogrus.func2({0x134a6b8, 0xc010c94a80}, 0xc01b824140)\n\tgithub.com/flashbots/go-utils@v0.11.0/httplogger/httplogger.go:139 +0xfc\nnet/http.HandlerFunc.ServeHTTP(0x0?, {0x134a6b8?, 0xc010c94a80?}, 0x4?)\n\tnet/http/server.go:2294 +0x29\ngithub.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1({0x1348c18, 0xc011ceb420}, 0xc01b824140)\n\tgithub.com/NYTimes/gziphandler@v1.1.1/gzip.go:336 +0x275\nnet/http.HandlerFunc.ServeHTTP(0x41b405?, {0x1348c18?, 0xc011ceb420?}, 0xc011ceb401?)\n\tnet/http/server.go:2294 +0x29\nnet/http.serverHandler.ServeHTTP({0x13456b8?}, {0x1348c18?, 0xc011ceb420?}, 0x6?)\n\tnet/http/server.go:3301 +0x8e\nnet/http.(*conn).serve(0xc01bcf6990, {0x134c458, 0xc00023a390})\n\tnet/http/server.go:2102 +0x625\ncreated by net/http.(*Server).Serve in goroutine 1\n\tnet/http/server.go:3454 +0x485\n",
  "version": ""
}

what is clear from the stacktrace is, payload api is panic-ing for signed blinded block requests (the most critical one!) but only while trying to verify a signature for a deneb block. post-fork, we shouldn't be getting any of those.

it seems in verifyBlockSignature while computing signingData.HashTreeRoot() we blow up. since we're post fork this seems easy enough to solve by checking if the payload is an electra payload and otherwise rejecting the request entirely.

why the panic happens is not clear to us.

leading explanation for us is that some old clients are still sending Deneb payloads. we particularly noted some DVT clusters sent with mev-boost 1.7 user agents but this seemed a partial explanation at best.

Expected behavior

1. to reject payloads only valid pre-fork, post-fork would be nice
2. to not panic when verifying old signatures

Which version are you using:
we run a slightly modified forked version. although the above seems unlikely due to our modifications it could be. would be interesting if other relays, specifically those running this code saw the same error.

Start in services/api/utils.go at verifyBlockSignature, then trace its callers in service.go through checkProposerSignature and handleGetPayload. Reproduce or inspect signed blinded-block verification for Deneb requests and determine the existing API response path. Done means pre-fork payloads are rejected without a panic during signature verification.