envoyproxy / envoyproxy/envoy

Integration test downstream client crashes when a long send is interrupted with an early close

Open
#17,753 2 comments 0 reactions 0 assignees View on GitHub
area/ext_proc bug help wanted
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 20h
Merged PRs (30d)
437

Description

Discussed this crash with @htuch before making public

*Title*: *Integration test downstream client crashes when a long send is interrupted with an early close*

*Description*:
I encountered the following crash while fuzzing the ext_proc filter.

If the downstream client is making a large POST request in chunks and Envoy responds before the downstream client is finished sending, a seg fault occurs.

Adding the following test case to ext_proc's [streaming_integration_test](https://github.com/envoyproxy/envoy/blob/main/test/extensions/filters/http/ext_proc/streaming_integration_test.cc) can recreate this scenario:

```
// Perform a long chunked send while the external process immediately responds to the headers
TEST_P(StreamingIntegrationTest, LongSendCrash) {
const uint32_t num_chunks = 150;
const uint32_t chunk_size = 100;

test_processor_.start(
[](grpc::ServerReaderWriter* stream) {
ProcessingRequest header_req;
ASSERT_TRUE(stream->Read(&header_req));
ASSERT_TRUE(header_req.has_request_headers());

ProcessingResponse header_resp;
auto* immediate = header_resp.mutable_immediate_response();
immediate->mutable_status()->set_code(envoy::type::v3::StatusCode::Unauthorized);
immediate->set_body("{\"reason\": \"Not authorized\"}");
immediate->set_details("Failed because you are not authorized");
stream->Write(header_resp);
});

initializeConfig();
HttpIntegrationTest::initialize();
sendPostRequest(num_chunks, chunk_size, absl::nullopt);

// TODO skipping checks of response
}

```

*Additional Information*:
I'm not sure if this is specific to ext_proc.

The test is fairly flaky, you can increase the `num_chunks` value if the issue isn't being recreated on your machine.

Further, I could only recreate the crash after executing all tests in the test suite.

*Logs:*
I have provided [truncated.log](https://github.com/envoyproxy/envoy/files/7003795/truncated.log) which shows the crash, but I removed the output from the other tests in the suite.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.