Failed to collect receipts in exchange for query fee voucher - for snapshot subgraph

at least 2 indexers (us and datanexus, and i bet there is more) have issues collecting query fees from snapshot subgraph (its a specific thing given that all others went through and we closed 37 allocations total):

Apr 17 10:06:52 ind03-grt-m graph-agent.sh[2626653]: {"level":20,"time":1681718812773,"pid":2626653,"hostname":"ind03-grt-m","name":"IndexerAgent","component":"AllocationReceiptCollector","allocation":"0xba3321f2cF4C79bF6C61B156B356d9433F54723d","partialVouchers":[{"allocation":"0xba3321f2cf4c79bf6c61b156b356d9433f54723d","fees":"5155882676354951903878","receipt_id_max":"0xf2b105541569ace8a6aa9970444e06","receipt_id_min":"0x0001290d86602fc91fcc7842f0c99b","signature":"0xe81ba7adc96eb0adc11d3fbb90247caac5fbf568d2de780fb0a096b28033ac4045598a931b2e88449d2bc36123960cd2b66c15df76a5fb08d4727eb08dc6834c1c"},{"allocation":"0xba3321f2cf4c79bf6c61b156b356d9433f54723d","fees":"314803236281300371082","receipt_id_max":"0xfff8fe03877c61e0575e6663da8951","receipt_id_min":"0xf2b2bc3b1b8183869e23bb3d4d0a6e","signature":"0xd3c549dcd8047dc3070d373f2fc6f7c51f857ae0a880e85857fcc6897d7fd50775f51f17b606c9cecbf69edd6f26f916b8a68606e84db42305d6b1f480ae97e51b"}],"hexStringLength":"0xba3321f2cf4c79bf6c61b156b356d9433f54723d","msg":"obtainReceiptVoucher: For debugging"}

Apr 17 10:06:52 ind03-grt-m graph-agent.sh[2626653]: {"level":50,"time":1681718812824,"pid":2626653,"hostname":"ind03-grt-m","name":"IndexerAgent","component":"AllocationReceiptCollector","allocation":"0xba3321f2cF4C79bF6C61B156B356d9433F54723d","err":{"type":"IndexerError","message":"Failed to collect receipts in exchange for query fee voucher","stack":"IndexerError: Failed to collect receipts in exchange for query fee voucher\n at indexerError (/home/pops/indexer/packages/indexer-common/dist/errors.js:169:12)\n at AllocationReceiptCollector.<anonymous> (/home/pops/indexer/packages/indexer-common/dist/allocations/query-fees.js:298:147)\n at Generator.throw (<anonymous>)\n at rejected (/home/pops/indexer/packages/indexer-common/dist/allocations/query-fees.js:6:65)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)","code":"IE054","explanation":"https://github.com/graphprotocol/indexer/blob/main/docs/errors.md#ie054","cause":{"message":"Request failed with status code 400","stack":"Error: Request failed with status code 400\n at createError (/home/pops/indexer/node_modules/axios/lib/core/createError.js:16:15)\n at settle (/home/pops/indexer/node_modules/axios/lib/core/settle.js:17:12)\n at IncomingMessage.handleStreamEnd (/home/pops/indexer/node_modules/axios/lib/adapters/http.js:322:11)\n at IncomingMessage.emit (node:events:525:35)\n at IncomingMessage.emit (node:domain:489:12)\n at endReadableNT (node:internal/streams/readable:1358:12)\n at processTicksAndRejections (node:internal/process/task_queues:83:21)","config":{"transitional":{"silentJSONParsing":true,"forcedJSONParsing":true,"clarifyTimeoutError":false},"transformRequest":[null],"transformResponse":[null],"timeout":0,"xsrfCookieName":"XSRF-TOKEN","xsrfHeaderName":"X-XSRF-TOKEN","maxContentLength":-1,"maxBodyLength":-1,"headers":{"Accept":"application/json, text/plain, */*","Content-Type":"application/octet-stream","User-Agent":"axios/0.26.1","Content-Length":276},"method":"post","url":"https://gateway.network.thegraph.com/voucher","data":{}},"status":400}},"msg":"Failed to collect receipts in exchange for an on-chain query fee voucher"}

I would hope this is treated as high priority since its affecting our revenue (if this is lost, thats 5,4k grt worth of queries we did for free for snapshot...) and because this is what the graph is all about, and if this doesnt work, the graph doesnt work.

@fordN

Start in packages/indexer-common/dist/allocations/query-fees.js at AllocationReceiptCollector and consult packages/indexer-common/dist/errors.js for IE054. Reproduce the voucher request failure for the snapshot subgraph using the logged allocation details, then verify receipt collection succeeds and the 400 error no longer occurs.