gradle / gradle/develocity-build-validation-scripts

Support more use cases for validation

Open
#631 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Shell
Stars
115
Forks
23
Avg merge
1d 4h
Merged PRs (30d)
6

Description

It is sometimes the case that a user will want to customize the validation done by the scripts when running in an automated environment to something other than `executed_cacheable_tasks == 0`.

A common use case is the user has one or more known cache misses they want to exclude from the assertion. More recently we had a case where someone wanted to fail Gradle experiment 1 if any tasks re-executed, but this isn't available out-of-the-box.

What I recommend doing in these scenarios is:

1. Do not include `--fail-if-not-fully-cacheable` in the validation workflows
2. Add a follow up step to the job [that calls the API](https://docs.gradle.com/enterprise/api-manual/ref/2024.1.html#tag/Builds/operation/GetBuild) and performs the desired validation using a tool like `jq` to parse the response

This isn't complicated, but does require the user to parse the CSV in the `.data` folder for the Build Scan ID, build the request (including authorization if necessary), and make the HTTP request.

However, the scripts already call the API to grab the attributes and build cache performance build models to do its own assertions. We should consider additionally writing these responses out to the corresponding experiment directory under the `.data` folder such that anyone requiring a tailored validation could do so more easily.

We should also consider doing #619 at the same time as these both make the custom validation use case easier.

Finally, we should use this ticket to keep track of any other use cases.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the validation scripts that call the Develocity Build API and read the Build Scan ID from the CSV in the .data folder. Define the response files and experiment-directory layout before implementing; done means tailored validation can consume the saved API responses without repeating request and authorization setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
shell
Domain
testing-qa, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.