GoogleCloudPlatform / GoogleCloudPlatform/functions-framework-conformance

Flakey tests - Validate healthy server startup before sending requests

Open
#72 0 comments 0 reactions 0 assignees View on GitHub
enhancement P2
Dominant language
Go
Stars
22
Forks
14
PR merge metrics
No merged PRs in 30d

Description

## Problem
The current validation logic starts up the functions framework servers, sleeps for a preconfigured time, and then attempts to send a request to the server.

1. Server start: https://github.com/GoogleCloudPlatform/functions-framework-conformance/blob/bb58df61a7f84c9ad865f21f8cbee8cacb01809d/client/validate.go#L76
2. Sleep (buildpack-based): https://github.com/GoogleCloudPlatform/functions-framework-conformance/blob/80ce4da7113a9d7140548b065e9cf1f91e4d32ed/client/buildpacks.go#L135
Sleep (local server): https://github.com/GoogleCloudPlatform/functions-framework-conformance/blob/bb58df61a7f84c9ad865f21f8cbee8cacb01809d/client/local.go#L53
3. Request is sent: https://github.com/GoogleCloudPlatform/functions-framework-conformance/blob/bb58df61a7f84c9ad865f21f8cbee8cacb01809d/client/validate.go#L85

The sleep time is configured by the client's `startDelay` flag and is configured per conformance test. If it's not configured long enough, it's possible that the server is not started up by the time the HTTP request is sent and the test fails or becomes flakey.

`startDelay` also forces a minimum wait time per test and might be adding unnecessary wait time.

Note that "server start" here is whatever command is passed to the conformance test client's `--cmd` flag and could be doing more work than just starting up the functions framework server, making it harder to choose the correct `startDelay`.

## Suggestion
It would be better if the `runValidation` function checked for "health status" by seeing if there is something serving on `localhost:8080` before sending the validating HTTP request instead of having each test guess the minimum required `startDelay`. That would eliminate the need to "guess" the correct start delay needed across different tests and repos.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.