etcd-io / etcd-io/etcd

Strictly define how JSONs are separated in GRPC gateway

Open
#11,372 39 comments 3 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
52.3k
Forks
10.5k
Avg merge
2d 21h
Merged PRs (30d)
43

Description

I use GRPC gateway to access etcd watching functionality. Namely, https://github.com/etcd-io/etcd/blob/master/Documentation/dev-guide/api_grpc_gateway.md#watch-keys .

But I don't have any specifications on how should I separate JSONs in the response stream since it is not documented anywhere.

Basically I have 3 ways:

1. HTTP chunk = message
This fails at least in Fedora 31 on high load because sometimes bare newline `\n` appears as a separate chunk. In order to fix you probably should first concatenate JSON with `\n` before calling to `Write()`. Also, intermediate proxies may re-chunk stream.

2. Messages are separated using strictly one newline.
It is not documented as well, and comment in GRPC gateway states that newline is added just to simplify debugging. Somewhere in the net, I found that other software uses a double newline for that purpose (allow to divide messages containing formatted JSON). Currently, I choose this way.

3. Use JSON finite state machine to detect end of a message.
This way will work in any case, but technically this requires additional libraries for the majority of programming languages.

So, Please strictly document exact behavior (protocol) which should be considered for implementing JSON stream parser.

Original issue:
https://github.com/grpc-ecosystem/grpc-gateway/issues/1086

Contributor guide

Open the contributing guide

Research direction

Start with the Watch Keys section of Documentation/dev-guide/api_grpc_gateway.md and review the linked grpc-gateway issue 1086. Determine the exact framing behavior for streamed JSON responses, including whether HTTP chunking or newline separation is normative. Done means the protocol and client parsing expectations are documented unambiguously.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grpc
Domain
api, distributed-systems, documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.