GoogleCloudPlatform / GoogleCloudPlatform/functions-framework-conformance
Flakey tests - Validate healthy server startup before sending requests
- Dominant language
- Go
- Stars
- 22
- Forks
- 14
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
The current validation logic starts up the functions framework servers, sleeps for a preconfigured time, and then attempts to send a request to the server.
1. Server start: https://github.com/GoogleCloudPlatform/functions-framework-conformance/blob/bb58df61a7f84c9ad865f21f8cbee8cacb01809d/client/validate.go#L76
2. Sleep (buildpack-based): https://github.com/GoogleCloudPlatform/functions-framework-conformance/blob/80ce4da7113a9d7140548b065e9cf1f91e4d32ed/client/buildpacks.go#L135
Sleep (local server): https://github.com/GoogleCloudPlatform/functions-framework-conformance/blob/bb58df61a7f84c9ad865f21f8cbee8cacb01809d/client/local.go#L53
3. Request is sent: https://github.com/GoogleCloudPlatform/functions-framework-conformance/blob/bb58df61a7f84c9ad865f21f8cbee8cacb01809d/client/validate.go#L85
The sleep time is configured by the client's `startDelay` flag and is configured per conformance test. If it's not configured long enough, it's possible that the server is not started up by the time the HTTP request is sent and the test fails or becomes flakey.
`startDelay` also forces a minimum wait time per test and might be adding unnecessary wait time.
Note that "server start" here is whatever command is passed to the conformance test client's `--cmd` flag and could be doing more work than just starting up the functions framework server, making it harder to choose the correct `startDelay`.
## Suggestion
It would be better if the `runValidation` function checked for "health status" by seeing if there is something serving on `localhost:8080` before sending the validating HTTP request instead of having each test guess the minimum required `startDelay`. That would eliminate the need to "guess" the correct start delay needed across different tests and repos.
Contributor guide
Assessment
This issue has not been assessed yet.