apache / apache/opendal

bug: etcd reads larger than 4 MiB fail at the client decoding limit

Open
#8,164 0 comments 1 reaction 0 assignees View on GitHub
bindings/dotnet bug releases-note/fix services/etcd
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

## OpenDAL version or commit

PR #8119 head `b838b5fadb8eb5e53b77ae75bf574eef2914488e`
(OpenDAL 0.58.2 in the reproduced CI job).

## Affected service or component

`services/etcd` against etcd v3.6.4.

## Describe the bug

The etcd service can write an 8 MiB value when the server allows it, but reading
the value fails because etcd-client 0.19.0 applies a default 4 MiB gRPC decoding
limit to KV responses.

This is a client response-size limit, not a write-size limit, so
`Capability::write_total_max_size` cannot describe it correctly.

## Steps to Reproduce

Start the repository's etcd fixture:

```bash
docker compose -f fixtures/etcd/docker-compose-standalone.yml up -d --wait

export OPENDAL_ETCD_ENDPOINTS=http://127.0.0.1:2379
export OPENDAL_ETCD_ROOT=/tmp/opendal
export OPENDAL_TEST=etcd

cd bindings/dotnet
cargo build
dotnet test -f net10.0 \
--filter 'FullyQualifiedName~ReadBehavior_LargePayload_RoundTripsExactly'
```

The fixture configures etcd with `--max-request-bytes=10485760`.

The failure is also reproduced in:
https://github.com/apache/opendal/actions/runs/32733492592/job/97459019848

## Actual Behavior

Writing the 8 MiB value succeeds. Reading it fails with:

```text
decoded message length too large: found 8388766 bytes, the limit is: 4194304 bytes
```

## Expected Behavior

The 8 MiB round trip succeeds when the etcd server is configured to accept the
payload and the client decoding limit is configured accordingly.

## Additional Context

etcd-client exposes `KvClient::max_decoding_message_size`, but OpenDAL's etcd
configuration currently provides no way to set it.

A fix should:

- Add an explicit etcd client decoding-size configuration without defaulting to
an unbounded response size.
- Apply the configured limit to KV reads.
- Configure the repository's etcd fixture with a limit large enough for the
existing 8 MiB behavior test.
- Remove the temporary etcd-specific skip after the behavior test passes.

Contributor guide

Open the contributing guide

Research direction

Start in services/etcd and inspect the etcd client setup, then run the repository fixture from fixtures/etcd/docker-compose-standalone.yml and the ReadBehavior_LargePayload_RoundTripsExactly test. Add a bounded decoding-size configuration, apply it to KV reads, configure the fixture for the 8 MiB payload, and remove the temporary etcd-specific skip when the round trip passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, rust
Domain
distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.