getsentry / getsentry/sentry-dotnet

Fix or remove flaky device tests

Open
#3,649 0 comments 0 reactions 0 assignees View on GitHub
.NET Repository Maintenance Tests
Dominant language
C#
Stars
770
Forks
248
Avg merge
3d 4h
Merged PRs (30d)
49

Description

We have disabled a number of tests on iOS and Android.

In cases where these are flaky both on CI and on a local dev machine, these are disabled with an `#if` directive:
```csharp
#if __IOS__
Skip.If(true, "Flaky on iOS");
#endif
```

When they're only flaky when run in CI, we check for an environment variable instead:
```csharp
Skip.If(TestEnvironment.IsGitHubActions, "This test is flaky in CI");
```

Ultimately we need to **_either_** work out why the tests fail and fix the code/environment/tests **_or_** simply remove the tests. There's no point in having tests that we don't trust in the repository. Preferably we'd fix the tests though.

## Flaky Tests on iOS

#### Logging Tests

https://github.com/getsentry/sentry-dotnet/blob/0acaff17253c1d1e3c572f0e28c779ced5987f5f/test/Sentry.Extensions.Logging.Tests/LoggingTests.cs#L15-L19

https://github.com/getsentry/sentry-dotnet/blob/0acaff17253c1d1e3c572f0e28c779ced5987f5f/test/Sentry.Extensions.Logging.Tests/LoggingTests.cs#L61-L65

The error message for both of the above tests is similar:

```
NSubstitute.Exceptions.ReceivedCallsException : Expected to receive exactly 1 call matching:
EnqueueEnvelope(e => (e.Items.Select(i => i.Payload).OfType().Select(i => i.Source).OfType().Single().Breadcrumbs.SingleOrDefault(b => ((Convert(b.Level, Int32) == Convert(value(Sentry.Extensions.Logging.Tests.LoggingTests+<>c__DisplayClass1_0).logLevel.ToBreadcrumbLevel(), Int32)) AndAlso (b.Message == "test message"))) != null))
Actually received no matching calls.
Received 1 non-matching call (non-matching arguments indicated with '*' characters):
EnqueueEnvelope(*Envelope*)
at NSubstitute.Core.ReceivedCallsExceptionThrower.Throw(ICallSpecification callSpecification, IEnumerable`1 matchingCalls, IEnumerable`1 nonMatchingCalls, Quantity requiredQuantity)
at NSubstitute.Routing.Handlers.CheckReceivedCallsHandler.Handle(ICall call)
at NSubstitute.Routing.Route.Handle(ICall call)
at NSubstitute.Core.CallRouter.Route(ICall call)
at NSubstitute.Proxies.CastleDynamicProxy.CastleForwardingInterceptor.Intercept(IInvocation invocation)
at Castle.DynamicProxy.AbstractInvocation.Proceed()
at NSubstitute.Proxies.CastleDynamicProxy.ProxyIdInterceptor.Intercept(IInvocation invocation)
at Castle.DynamicProxy.AbstractInvocation.Proceed()
at Castle.Proxies.ObjectProxy_3.EnqueueEnvelope(Envelope envelope)
at Sentry.Extensions.Logging.Tests.LoggingTests.Log_AddsBreadcrumb(LogLevel logLevel)
at System.Reflection.MethodInvoker.InterpretedInvoke(Object obj, Span`1 args, BindingFlags invokeAttr)
```

#### Maui Screenshot Tests

https://github.com/getsentry/sentry-dotnet/blob/0acaff17253c1d1e3c572f0e28c779ced5987f5f/test/Sentry.Maui.Tests/SentryMauiScreenshotTests.cs#L36-L42

https://github.com/getsentry/sentry-dotnet/blob/0acaff17253c1d1e3c572f0e28c779ced5987f5f/test/Sentry.Maui.Tests/SentryMauiScreenshotTests.cs#L73-L78

The error messages is similar for both of the above tests:

```


.]]>



```

## Flaky Tests on Android

We're not getting much useful information back about why the tests are failing on Android. Xharness simply says that the tests timed out after 900 seconds.

```
XHarness exit code: 70 (TIMED_OUT)
```

There are some adb logs but it's very hard to get any useful information out of these.

#### Logging Tests

https://github.com/getsentry/sentry-dotnet/blob/0acaff17253c1d1e3c572f0e28c779ced5987f5f/test/Sentry.Extensions.Logging.Tests/LoggingTests.cs#L15-L20

#### Hub Tests

https://github.com/getsentry/sentry-dotnet/blob/0acaff17253c1d1e3c572f0e28c779ced5987f5f/test/Sentry.Tests/HubTests.cs#L1137-L1139

#### SDK Tests

`https://github.com/getsentry/sentry-dotnet/blob/0acaff17253c1d1e3c572f0e28c779ced5987f5f/test/Sentry.Tests/SentrySdkTests.cs#L430-L432`

### Android API Levels

On Android, it appears the reliability of tests is impacted by the API Level that we're targeting.

We used to test against API levels 27 -31, for example, but the tests would almost invariably fail in one or more of those API levels, so we dropped back to testing against just 27 and 31 in [this PR](https://github.com/getsentry/sentry-dotnet/commit/815f2215ecc4155593a704c7560b1677c234025f).

In PR #3628 the tests were often passing against API level 27 but failing against API level 33.

This may be related to [Restrictions on non-SDK interfaces](https://developer.android.com/guide/app-compatibility/restrictions-non-sdk-interfaces) but, if so, I don't understand how.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.