chaoss / chaoss/disclosure

CLI & Action: disclosure text lacks --min-confidence support (breaks documented examples and causes false positives in GitHub Actions)

Open
#102 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
26
Forks
13
Avg merge
6d 18h
Merged PRs (30d)
3

Description

### Description

There is a significant CLI UX, documentation, and functional gap between `disclosure scan` and `disclosure text` regarding confidence filtering and flag naming:

1. **Documented CLI help example fails with `unknown flag: --min-confidence`:**
In [`cmd/cmd.go`](https://github.com/chaoss/disclosure/blob/main/cmd/cmd.go#L273) (lines 273 and 282), `disclosure text`'s own help text (`Long` description and `Example`) instructs users:
```sh
cat comment.txt | disclosure text --min-confidence=medium
```
However, `textCommand` does not register a `--min-confidence` flag. Running this documented command immediately fails:
```
Error: unknown flag: --min-confidence
```

2. **GitHub Action `min-confidence` is bypassed for PR bodies, causing false positives:**
In [`action/action.yml`](https://github.com/chaoss/disclosure/blob/main/action/action.yml#L18-L23), the `min-confidence` input documentation states:
```yaml
# set min-confidence to medium or above if your PR template has AI tools mentioned in comments etc.
# this will help avoid false positives especially when using AI use checkboxes.
```
However, because `disclosure text` lacks `--min-confidence` support, [`action/action.yml`](https://github.com/chaoss/disclosure/blob/main/action/action.yml#L97-L103) invokes `ai-detection text` without `--min-confidence`:
```bash
TEXT_REPORT=$(echo "${PR_BODY}" | ${{ runner.temp }}/ai-detection text \
--format=json ${CHECKBOX_DETECTION_FLAG} \
--cb-disclosed-ai="${CHECKBOX_LABEL_AI_USED}" \
--cb-disclosed-noai="${CHECKBOX_LABEL_AI_NOT_USED}") || TEXT_EXIT=$?
if [ "${TEXT_EXIT:-0}" = "1" ]; then
AI_DETECTED=true
fi
```
Whenever a PR body mentions any tool name (e.g., in PR template instructions, comments, or disclosure checkboxes), `toolmention` generates low-confidence findings (score 20). Because `textCommand` does not filter by confidence, any finding causes an exit code of `1` (`ExitAI`).
Consequently, the action sets `ai-detected=true` and flags/labels the PR as AI-detected, completely bypassing the user-configured `min-confidence: 'medium'` or `'high'` threshold.

3. **Documented CLI usage flags in `README.md` do not exist:**
In [`README.md`](https://github.com/chaoss/disclosure/blob/main/README.md#cli-usage) (lines 30-35):
```
disclosure text \
[--format=json|text] [--input=FILE|-] \
[--checkbox-label-ai-used="AI was used"] \
[--checkbox-label-ai-not-used="AI was not used"]
```
- Flags `--checkbox-label-ai-used` and `--checkbox-label-ai-not-used` do not exist in the CLI (renamed in PR #81 commit `45a48c0` to `--cb-disclosed-ai` and `--cb-disclosed-noai`). Passing them fails with `unknown flag: --checkbox-label-ai-used`.
- The `--enable-checkbox-detection` flag is omitted from the synopsis.
- In `cmd/cmd.go` line 289, the example snippet is missing a line continuation backslash `\` after `--enable-checkbox-detection`.

---

### Steps to Reproduce

1. Execute the command documented in `disclosure text --help`:
```sh
echo "Claude was mentioned" | disclosure text --min-confidence=medium
```
**Output:**
```
Error: unknown flag: --min-confidence
```

2. Configure the GitHub Action with `min-confidence: 'medium'` and `scan-pr-body: 'true'`:
- Open a PR containing a tool mention in its description (e.g. `Claude` in template instructions).
- Although the commit scan filters by `medium`, `disclosure text` outputs the low-confidence finding (score 20) and exits with `1`.
- The action sets `ai-detected=true` despite `min-confidence: 'medium'`.

---

### Expected Behavior

1. `disclosure text` should support `--min-confidence` (and `--confidence-levels`), filtering findings below the specified threshold. If all findings are below `min-confidence`, it should exit with `0` (`ExitNoAI`).
2. `action/action.yml` should pass `--min-confidence="${MIN_CONFIDENCE}"` to the text scan command so that PR body scans respect the configured confidence threshold.
3. Help text and `README.md` should accurately document the active flags (`--cb-disclosed-ai`, `--cb-disclosed-noai`, `--enable-checkbox-detection`, and `--min-confidence`).

---

### Proposed Solution

1. **`cmd/cmd.go`**:
- Add `--min-confidence` (and optionally `--confidence-levels`) flags to `textCommand`.
- Filter `findings` based on `minConf` before formatting output and setting `exitCode`.
- Add missing trailing backslash `\` in `cmd/cmd.go` line 289.
2. **`action/action.yml`**:
- Pass `--min-confidence="${MIN_CONFIDENCE}"` to the `ai-detection text` invocation.
3. **`README.md`**:
- Update `disclosure text` usage synopsis to reflect `--cb-disclosed-ai`, `--cb-disclosed-noai`, `--enable-checkbox-detection`, and `--min-confidence`.

Contributor guide

Open the contributing guide

Research direction

Start in cmd/cmd.go at textCommand and compare its flags, help text, and exit-code handling with the scan command. Then inspect the text invocation in action/action.yml and the CLI synopsis in README.md. Done means documented flags work, PR-body scans pass the configured confidence threshold, and the help and README examples match the active flag names.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, go
Domain
ci-cd, cli, documentation
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.