Strictly define how JSONs are separated in GRPC gateway
- Dominant language
- Go
- Stars
- 52.3k
- Forks
- 10.5k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 43
Description
I use GRPC gateway to access etcd watching functionality. Namely, https://github.com/etcd-io/etcd/blob/master/Documentation/dev-guide/api_grpc_gateway.md#watch-keys .
But I don't have any specifications on how should I separate JSONs in the response stream since it is not documented anywhere.
Basically I have 3 ways:
1. HTTP chunk = message
This fails at least in Fedora 31 on high load because sometimes bare newline `\n` appears as a separate chunk. In order to fix you probably should first concatenate JSON with `\n` before calling to `Write()`. Also, intermediate proxies may re-chunk stream.
2. Messages are separated using strictly one newline.
It is not documented as well, and comment in GRPC gateway states that newline is added just to simplify debugging. Somewhere in the net, I found that other software uses a double newline for that purpose (allow to divide messages containing formatted JSON). Currently, I choose this way.
3. Use JSON finite state machine to detect end of a message.
This way will work in any case, but technically this requires additional libraries for the majority of programming languages.
So, Please strictly document exact behavior (protocol) which should be considered for implementing JSON stream parser.
Original issue:
https://github.com/grpc-ecosystem/grpc-gateway/issues/1086
Contributor guide
Research direction
Start with the Watch Keys section of Documentation/dev-guide/api_grpc_gateway.md and review the linked grpc-gateway issue 1086. Determine the exact framing behavior for streamed JSON responses, including whether HTTP chunking or newline separation is normative. Done means the protocol and client parsing expectations are documented unambiguously.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, grpc
- Domain
- api, distributed-systems, documentation
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100