I suggest adding an "Accuracy" property to DiagnosticRecord.

Suggestion

Add an "Accuracy" property to DiagnosticRecord that will state the probability of the diagnostic being correct.

Benefits for the Users

Users can filter and sort diagnostics based on their accuracy.

    invoke-scriptanalyzer |
        where {$_.accuracy -gt 0.75} |
        sort rulename, accuracy -descending

Benefits for the Rule Writers

Rule writers can introduce new rules sooner by introducing rules with a low accuracy, and then later increase the accuracy as the rule is refined.

For example, the "UseShouldProcessForStateChangingFunctions" rule right now is based only on the function name, so the rule can be assigned with an accuracy of, let's say, 0.25. If the rule is improved with better heuristics, then its accuracy can be increased. If the default of Invoke-ScriptAnalyzer doesn't show diagnostics with an accuracy of less than 0.50, then users won't have to suppress the "UseShouldProcessForStateChangingFunctions" while the rule is still in its initial stages.

This allows rule writers to get feedback on their rules while the rules are still being refined without adding noise to Invoke-ScriptAnalyzer results.

Benefits for the Rules

The rules can have multiple diagnostic accuracies.

For example, the "PossibleIncorrectComparisonWithNull" will be more useful if it can output diagnostics with different accuracies based on context.

A line such as:

    if ($a -ne $null)

should output a diagnostic for this rule with a high accuracy because "$a -ne $null" is very likely to be a compare operation instead of a filter operation.

However, a line such as:

    $a = $b -ne $null

should output a diagnostic for this rule with a low accuracy because "$b -ne $null" can either be a compare operation or a filter operation, and it will be very difficult to assert that the user violated the rule.

Now, the rule doesn't have to be hardcoded with the compromise of when to generate diagnostics because it can generate all diagnostics with different accuracy levels.

Start by tracing DiagnosticRecord and the Invoke-ScriptAnalyzer output path; the issue does not name source files or tests. Determine how accuracy should be represented, produced by rules, and exposed for filtering and sorting. Done means the property and its semantics are consistently supported across the proposed diagnostic scenarios.