firebase / firebase/firebase-tools
Feature Request: CLI/API to get App Testing Agent test run status and results
- Dominant language
- TypeScript
- Stars
- 4.5k
- Forks
- 1.3k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 84
Description
## Feature Request
### Description
The App Testing Agent (AI-powered testing) for Firebase App Distribution currently has no CLI command or REST API to programmatically retrieve test run status and results. This prevents integration into CI/CD pipelines for automated quality gates.
### Current State
- `firebase appdistribution:testcases:import` - ✅ Works for importing test cases
- `firebase appdistribution:distribute --test-devices=... --test-case-ids=...` - ✅ Works for triggering tests
- **Getting test results** - ❌ Only available via Firebase Console UI
### Gradle Plugin Investigation
We investigated the Gradle plugin as a potential workaround. The `firebaseAppDistribution` block supports a `testNonBlocking` parameter:
```kotlin
firebaseAppDistribution {
testDevices = "model=shiba,version=34,locale=en,orientation=portrait"
testCases = "prescriptive-workouts"
testNonBlocking = true // or false?
}
```
**Documentation gap:** The [Gradle distribution docs](https://firebase.google.com/docs/app-distribution/android/distribute-gradle) only describe `testNonBlocking = true`:
> "Run automated tests asynchronously. Visit the Firebase console for the automatic test results."
**What's NOT documented:**
- What happens when `testNonBlocking = false` (or omitted)?
- Does the Gradle task block until tests complete?
- Are exit codes returned (0 for pass, non-zero for fail)?
- What's the timeout behavior?
If `testNonBlocking = false` provided blocking behavior with exit codes, this would be a partial solution. But the behavior is undocumented.
### Requested Feature
Add CLI commands and/or REST API endpoints to:
1. **List test runs** for a release
```bash
firebase appdistribution:testruns:list --app --release
```
2. **Get test run status/results**
```bash
firebase appdistribution:testruns:get --app
```
Should return JSON with:
- Test case ID and name
- Status (pending, running, passed, failed)
- Device information
- Timestamp
- Link to detailed results
- (Optionally) Issues found, screenshots
3. **Wait for test completion** (for CI/CD blocking)
```bash
firebase appdistribution:testruns:wait --app --release --timeout 30m
```
Exit code 0 if all pass, non-zero if any fail.
### Use Case
We're using the App Testing Agent to run E2E tests defined in YAML. We want to:
1. Build APK in CI
2. Distribute to App Testing Agent
3. **Wait for tests to complete and get results** ← Currently impossible
4. Fail the CI pipeline if tests fail
### Workaround Attempts
- Firebase App Distribution REST API (`firebaseappdistribution.googleapis.com`) - No test result endpoints
- Firebase Test Lab gcloud commands - Different service, doesn't cover App Testing Agent
- Cloud Functions triggers - App Testing Agent completion events not documented
- Gradle plugin `testNonBlocking` - Behavior undocumented (see above)
### Environment
- Firebase CLI version: 13.x
- Feature: App Testing Agent (Preview)
### Additional Context
The App Testing Agent is a powerful feature, but without programmatic result access, it can only be used for manual/ad-hoc testing rather than automated quality gates. Adding this would significantly increase adoption for teams with CI/CD pipelines.
Could the Firebase team also clarify the `testNonBlocking = false` behavior in the Gradle plugin? If it already supports blocking with exit codes, documenting this would help teams in the interim.
Contributor guide
Research direction
Start by reviewing the existing firebase appdistribution:testcases:import and firebase appdistribution:distribute commands, then compare the firebaseappdistribution Gradle documentation for testNonBlocking. The issue names no files or tests; done would require an agreed CLI/API design that lists test runs, retrieves results, and supports waiting with CI exit behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- api, cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100