android / android/nowinandroid

[Bug]: ScrollForYouFeedBenchmark tests are failing with Macrobenchmark library v1.2.0-alpha15

Open
#794 1 comment 1 reaction 2 assignees Claimed by @DirajHS View on GitHub
bug
Dominant language
Kotlin
Stars
21.8k
Forks
4.6k
Avg merge
19h 20m
Merged PRs (30d)
2

Description

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Is there a StackOverflow question about this issue?

- [X] I have searched StackOverflow

### What happened?

`ScrollForYouFeedBenchmark` tests are failing with the Macrobenchmark library v1.2.0-alpha15. It looks like an undocumented behavior change has been [released](https://developer.android.com/jetpack/androidx/releases/benchmark#version_12_2) on how `setupBlock` and `measureBlock` runs with `startupMode` set to `StartupMode.COLD`. With the current stable [version v1.1.1](https://developer.android.com/jetpack/androidx/releases/benchmark#version_111_3) , `measureBlock` runs after the Activity is started implicitly by the Android system. In the newer version v1.2.0-xxx , we must start the Activity before running the `measureBlock` block

### With Macrobenchmark v1.1.1:
```kotlin
private fun scrollFeed(compilationMode: CompilationMode) = benchmarkRule.measureRepeated(
// ...
startupMode = StartupMode.COLD,
setupBlock = {
// Start the app
pressHome()
startActivityAndWait()
},
) {
// ... measureBlock
}
```

### With Macrobenchmark v1.2.0-alpha15:
```kotlin
private fun scrollFeed(compilationMode: CompilationMode) = benchmarkRule.measureRepeated(
// ...
startupMode = StartupMode.COLD,
setupBlock = {
// Start the app
pressHome()
startActivityAndWait()
},
) {
startActivityAndWait()
// ... measureBlock
}
```

`StartupMode.HOT` & `StartupMode.WARM` do not have any changes and work as expected.

### Relevant logcat output

```shell
java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.Object androidx.test.uiautomator.UiObject2.wait(androidx.test.uiautomator.UiObject2Condition, long)' on a null object reference
at com.google.samples.apps.nowinandroid.foryou.ForYouActionsKt.forYouWaitForContent(ForYouActions.kt:32)
at com.google.samples.apps.nowinandroid.foryou.ScrollForYouFeedBenchmark$scrollFeed$2.invoke(ScrollForYouFeedBenchmark.kt:52)
at com.google.samples.apps.nowinandroid.foryou.ScrollForYouFeedBenchmark$scrollFeed$2.invoke(ScrollForYouFeedBenchmark.kt:40)
at androidx.benchmark.macro.MacrobenchmarkKt$macrobenchmark$measurements$1$1$tracePath$1.invoke(Macrobenchmark.kt:260)
at androidx.benchmark.macro.MacrobenchmarkKt$macrobenchmark$measurements$1$1$tracePath$1.invoke(Macrobenchmark.kt:218)
at androidx.benchmark.perfetto.PerfettoCaptureWrapper.record(PerfettoCaptureWrapper.kt:129)
at androidx.benchmark.perfetto.PerfettoCaptureWrapper.record$default(PerfettoCaptureWrapper.kt:93)
at androidx.benchmark.macro.MacrobenchmarkKt$macrobenchmark$measurements$1.invoke(Macrobenchmark.kt:218)
at androidx.benchmark.macro.MacrobenchmarkKt$macrobenchmark$measurements$1.invoke(Macrobenchmark.kt:203)
at androidx.benchmark.perfetto.PerfettoTraceProcessor$Companion.runServer(PerfettoTraceProcessor.kt:105)
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmark(Macrobenchmark.kt:203)
at androidx.benchmark.macro.MacrobenchmarkKt.macrobenchmarkWithStartupMode(Macrobenchmark.kt:388)
at androidx.benchmark.macro.junit4.MacrobenchmarkRule.measureRepeated(MacrobenchmarkRule.kt:107)
at com.google.samples.apps.nowinandroid.foryou.ScrollForYouFeedBenchmark.scrollFeed(ScrollForYouFeedBenchmark.kt:40)
at com.google.samples.apps.nowinandroid.foryou.ScrollForYouFeedBenchmark.scrollFeedCompilationNone(ScrollForYouFeedBenchmark.kt:35)
```

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.