googleapis / googleapis/google-cloud-cpp

Make integration tests more tolerant of transient failures

Open
#10,413 30 comments 0 reactions 0 assignees View on GitHub
cpp: flake type: cleanup
Dominant language
C++
Stars
659
Forks
462
Avg merge
1d 2h
Merged PRs (30d)
89

Description

Our integration tests often perform "non-idempotent" operations, but these can fail even if our code is working fine. A typical test may have this structure:

```cc
auto create = client.CreateFoo(....);
ASSERT_STATUS_OK(create);

auto update = client.UpdateFoo(...);
ASSERT_STATUS_OK(update);

auto del = client.DeleteFoo(...);
ASSERT_STATUS_OK(del);
```

None of those operations are retried by default, as the operations are non-idempotent. I think we could rewirte the tests as follows:

```cc
auto create = client.CreateFoo(... Options{}.set(RetryEverything()));
ASSERT_THAT(create, StatusIs(AnyOf(StatusCode::kOk, StatusCode::kAlreadyExists)));
```

Or if we don't want to retry the operation, we can write it as:

```cc
auto create = client.CreateFoo(...);
if (IsTransient(...)) GTEST_SKIP() << create.status();
```

This probably requires a mini-design doc, and will require breaking down the work per test.

For now I am going to consolidate all existing "transient error on non-idempotent operation" flakes to this one.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.