github-community-projects / github-community-projects/safe-settings

Inconsistent diffs and behaviour

Open
#693 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
JavaScript
Stars
921
Forks
226
Avg merge
18h 3m
Merged PRs (30d)
14

Description

## Problem Description

We've been using safe-settings to manage a bunch of our repos now and for the most part it's been great 🎉 Unfortunately, we've encountered a couple of bugs/inconsistencies while using it, which make it hard to be confident that the right settings are applied from an infrastructure-as-code perspective.

### What is actually happening

A few things we've hit:

#### Inaccurate diffs

Occasionally we get inaccurate or unexpected diffs on the PR comments. For example:

![Screen Shot 2024-09-27 at 17 17 09](https://github.com/user-attachments/assets/831b4ed7-4a18-4608-b0c0-4872967c6534)

This was for a change where we didn't modify the `collaborators` setting.

#### Doesn’t handle deletion of settings very well
We are trying to replace our existing branch protection rules with rulesets, but removing/emptying the `branches` setting didn't actually do a delete of the branch protection rules as we expected.

```
branches: []

rulesets:
- name: default
target: branch
enforcement: active
...
```

Another case is where we tried to remove the `integration_id` on a status check context, but it was not actually removed:

```
- type: required_status_checks
parameters:
strict_required_status_checks_policy: true
required_status_checks:
- context: "my ci check"
integration_id: 1234567 # Removing this line didn't actually do anything
```

#### Missing context on PR comments and status check results

We also noticed that the PR comments and status checks reported by safe-settings are not always very informative or have the right context. See the following examples:

![image](https://github.com/user-attachments/assets/250e17cc-e553-4899-bd31-8c2f26a67169)
![image (1)](https://github.com/user-attachments/assets/6bd474d8-178d-4530-8e96-8567818c5afc)

Keen to know if these are genuine bugs or if there's something we've missed when setting up safe-settings! 🙏🏻

## Context

### Are you using the hosted instance of probot/settings or running your own?

We are running our own instance in a Kubernetes cluster

### If running your own instance, are you using it with github.com or GitHub Enterprise?

We are using it with standard github.com (Team plan)

#### Version of probot/settings

Helm chart version 2.1.11 (appVersion 2.1.2)

#### Version of GitHub Enterprise

N/A

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.