oasisprotocol / oasisprotocol/oasis-core
Revisit gRPC unexpected EOF error in daily tests
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 369
- Forks
- 151
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 6
Description
In the daily (12-hour long) E2E tests occasionally (about once in 10 runs?) a gRPC request in workload-queries fails with unexpected EOF. It seems to always happen during the GetTransactionsWithResults request.
Http2 trace logs indicate that client resets the stream with FLOW_CONTROL_ERROR:
2020/09/14 13:54:28 http2: Framer 0xc001df4540: wrote WINDOW_UPDATE len=4 (conn) incr=21661
2020/09/14 13:54:28 http2: Framer 0xc001df4540: wrote PING len=8 ping="\x02\x04\x10\x10\t\x0e\a\a"
2020/09/14 13:54:28 http2: Framer 0xc001df4540: wrote WINDOW_UPDATE len=4 (conn) incr=32768
2020/09/14 13:54:28 http2: Framer 0xc001df4540: wrote RST_STREAM stream=196089 len=4 ErrCode=FLOW_CONTROL_ERROR
2020/09/14 13:54:28 http2: Framer 0xc001df4540: wrote WINDOW_UPDATE len=4 (conn) incr=32768
{"caller":"grpc.go:242","err":"rpc error: code = Internal desc = unexpected EOF","level":"error","method":"/oasis-core.Consensus/GetTransactionsWithResults","module":"grpc/client","msg":"request failed","req_seq":2902768,"rsp":{"transactions":null,"results":null},"ts":"2020-09-14T13:54:28.779105914Z"}
No known way for reproducing this outside of the daily E2E test run (e.g. just doing large request/responses doesn't trigger this).
https://github.com/oasisprotocol/oasis-core/pull/3302 adds a workaround to the queries workload for this error, but this should be revisited and investigated further.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the workload-queries daily E2E path and the gRPC client log point at grpc.go:242, focusing on GetTransactionsWithResults and the FLOW_CONTROL_ERROR trace. Compare the workaround in PR 3302 with the failing request; done means identifying a reproducible cause or a justified fix, with the daily test no longer producing unexpected EOF.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, grpc
- Domain
- api, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100