0xMiden / 0xMiden/node

Response size estimates are off

Open
#2,310 4 comments 0 reactions 0 assignees View on GitHub
rpc store
Dominant language
Rust
Stars
104
Forks
138
Avg merge
1d 13h
Merged PRs (30d)
56

Description

`SyncTransactions` can return responses larger than the default tonic decode limit of 4 MB. One observed response from a pioneer was about 6 MB:

```rust
RpcError(RequestError {
endpoint: SyncTransactions,
error_kind: OutOfRange,
source: Status { code: OutOfRange,
message: "Error, decoded message length too large: found 6146702 bytes, the limit is:
4194304 bytes" }
})
```

Account ID was `0x6d37b2d4aedd697140338bb31c67e3`.

For pagination, the response uses the store's `size_in_bytes` estimate, which is obviously different than the actual protobuf-encoded response size. That estimate also undercounts the real payload: `fee` is not counted at all, unauthenticated input notes are estimated at 64 bytes but it ignores the note headers, etc. All this without counting the protobuf framing overhead.

I think @Dominik1999 also hit a similar issue on the `GetNotesById` endpoint, as part of the client's sync state process.

Overall, there are a couple of things to address:

- While the estimate is calculated based on the serialized raw structs, the payload may be filled too close to the 4 MB limit, which will cause the protobuf-encoded message to go over that due to the framing overhead
- Even so, it's clear some limits need to be reviewed because they may not be correctly counted (i.e., 4 MB vs 6 MB is a big enough difference)
- From the client, we do two things: we honor the node-communicated limits for parameters, and we keep the default 4 MB as the maximum-allowed payload size. We should probably bump this limit by 10-15% to allow for overhead if we can't enforce the limit correctly on the node.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.