apache / apache/arrow

[CI][FS][Azure] Azurite tests are flaking on `main`

Open
#40,121 2 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Continuous Integration Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Flaky failures like
```
C++ exception with description "Connection closed before getting full response or response is less than expected. Expected response length = 254. Read until now = 231" thrown in the test body.
2024-02-18T12:50:20.039Z ada6933e-9c33-47d2-86f6-29e9aa01f713 info: BlobStorageContextMiddleware: RequestMethod=DELETE RequestURL=http://127.0.0.1/devstoreaccount1/container?restype=container RequestHeaders:{"authorization":"SharedKey devstoreaccount1:hYh+JRj5cBYqdqOyM2wB3EZizQ/s2DiIoDI0CIF2EXM=","host":"127.0.0.1:10000","user-agent":"azsdk-cpp-storage-blobs/12.10.0-beta.1 (Linux 6.2.0-1019-azure x86_64 #19~22.04.1-Ubuntu SMP Wed Jan 10 22:57:03 UTC 2024)","x-ms-client-request-id":"be6819a2-72b8-4630-8eb0-4a88e7cb3061","x-ms-date":"Sun, 18 Feb 2024 12:50:20 GMT","x-ms-version":"2022-11-02"} ClientIP=127.0.0.1 Protocol=http HTTPVersion=1.1
```
I've seen the occur in different test cases and in different test suites.

Example failures:
I've seen one flake on `main`: https://github.com/apache/arrow/actions/runs/7915689559/job/21608061673
Flakes on my recent PRs:
https://github.com/apache/arrow/actions/runs/7951594516/job/21705210845?pr=40080
https://github.com/apache/arrow/actions/runs/7949050250/job/21699789831?pr=40080

### Component(s)

C++, Continuous Integration

Contributor guide

Open the contributing guide

Research direction

Start with the Azurite-backed C++ test suites and the linked GitHub Actions job logs, comparing the intermittent connection-closed failures across the reported runs. Identify the affected test entry points and reproduce the failure under CI conditions; done means the Azurite tests run reliably without these flaky response-length errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, cpp, github-actions
Domain
ci-cd, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.