ComplianceAsCode / ComplianceAsCode/content-test-filtering

RFE - Option to skip rules not applicable for RHEL

Open
#36 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
8
Forks
9
PR merge metrics
No merged PRs in 30d

Description

When reviewing a PR which changed rules only applicable to OL, though an explicit defition of [prodtype: ol7](https://github.com/ComplianceAsCode/content/blob/master/linux_os/guide/system/software/sap_host/accounts_authorized_local_users_sidadm_orasid/rule.yml#L3), it was noted that CI tests only relevant for RHEL and Fedora were executed and failed with the following output:
```
---------Bash Remediation Logs---------
ERROR - Rule 'xccdf_org.ssgproject.content_rule_accounts_authorized_local_users_sidadm_orasid' isn't present in benchmark 'xccdf_org.ssgproject.content_benchmark_RHEL-8' in '/tmp/ssgts-ds-gzmbdvrc'
ERROR - No tests found matching the rule ID(s) 'accounts_authorized_local_users_sidadm_orasid, java_jre_insecure_prompt_locked, java_jre_enable_jws, java_jre_security_revocation_check_locked, java_jre_validation_crl_locked, java_jre_validation_ocsp, java_jre_deployment_config_mandatory, java_jre_lock_untrusted_sources, java_jre_deployment_properties_exists, java_jre_validation_crl, java_jre_disable_untrusted_sources, java_jre_validation_ocsp_locked, java_jre_accepted_sites_exists, java_jre_unsigned_applications, java_jre_unsigned_applications_locked, java_jre_security_revocation_check, java_jre_blacklist_check_locked, java_jre_enable_jws_locked, java_jre_deployment_config_properties, java_jre_blacklist_check, java_jre_disable_untrusted_sources_locked, java_jre_lock_untrusted_sources_locked, java_jre_accepted_sites_properties, java_jre_insecure_prompt, java_jre_deployment_config_exists'
WARNING - Nothing has been tested!
```
The output is correct since the rule is indeed not available for RHEL8, as in this example.

In such cases, testing this rule in RHEL is not necessary since the result will always fail.
It would be good to have an option or other mechanism to not test rules in some products when the rules are not applicable for the respective product.

References:
PR: https://github.com/ComplianceAsCode/content/pull/9398
Tests CS8: https://github.com/ComplianceAsCode/content/runs/7998438537?check_suite_focus=true
Tests CS9: https://github.com/ComplianceAsCode/content/runs/7998421987?check_suite_focus=true

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the CI tests and the linked PR and test runs to understand how rules are currently selected for RHEL and Fedora. Done means rules that are not applicable to a product can be excluded from that product's tests without producing missing-rule failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.