CrowdStrike / CrowdStrike/csproto

Allow overriding the message size limit of 2GB

Open
#207 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement triage
Dominant language
Go
Stars
140
Forks
25
Avg merge
13h 3m
Merged PRs (30d)
3

Description

Title

Add a Decoder.SetMaxFieldLength() function to allow overriding the default limit of 2GB.

Description

The Decoder enforces a limit of 2GB on variable length fields like strings, bytes and nested messages.

We hit this limit for large snapshots in PowerDNS Lightning Stream as described in https://github.com/PowerDNS/lightningstream/issues/112

We propose adding a Decoder.SetMaxFieldLength() function to allow overriding this limit, while still retaining the default of 2GB when not overridden (PR incoming).

Additional Info

This limit is recommended by https://protobuf.dev/programming-guides/proto-limits/ as a way to guarantee interoperability.
It is also the only limit on that page that is not specified as being language dependent.

Go internally uses a 64-bit int for sizes on 64ibit platforms, and the actual length in the wire format is encoded in a varint that is decoded as a uint64 and can represent its full range, so there is no technical reason to impose a 2GB limit in a Go implementation.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the Decoder implementation and the existing 2GB limit for variable-length fields. Add the proposed SetMaxFieldLength entry point while preserving the default limit, then verify that strings, bytes, and nested messages honor an overridden value and retain the default when it is not set.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend-api-design
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.