ComplianceAsCode / ComplianceAsCode/content

Automatus struggles with profile-specific test scenarios

Open
#10,369 0 comments 0 reactions 0 assignees View on GitHub
Test Suite triaged
Dominant language
Shell
Stars
2.8k
Forks
828
Avg merge
3d 8m
Merged PRs (30d)
80

Description

#### Description of problem:
During his work on https://github.com/ComplianceAsCode/content/pull/10348, @maage has discovered that with some rules there are unexpected errors when Automatus runs in rule mode. This happens when testing rule `sssd_ldap_configure_tls_ca_dir` on a RHEL 8 virtual machine.

The trigger seems to be that the test scenarios for this rule have profile limitation in their headers but the rule isn't part of the given profile. In rule `sssd_ldap_configure_tls_ca_dir` all the test scenarios have `# profiles = xccdf_org.ssgproject.content_profile_stig` but the rule isn't part of RHEL 8 STIG profile. It is only part of RHEL 7 STIG profile and Oracle Linux 7 profile. But still the rule is present in RHEL 8 data stream.

As a consequence of the rule not being present in the profile, for the .fail.sh test scenarios, Automatus hits the issue in OpenSCAP https://github.com/OpenSCAP/openscap/issues/1963 that means that it generates an empty Ansible Playbook and then Automatus tries to apply this empty Ansible Playbook which doesn't remediate anything and then the final scan isn't fine.

The main question is the expected behavior for the profile-specific scenarios in a situation when the given rule isn't part of the profile but exists in the given SCAP source data stream.

#### SCAP Security Guide Version:
current upstream as of 2022-03-27 as of HEAD 2c185020f791e79b2621f2df98b67271de708efe

#### Operating System Version:
F 37

#### Steps to Reproduce:

1. python3 tests/automatus.py rule --libvirt qemu:///system ssgts_rhel8 --remediate-using ansible --dontclean sssd_ldap_configure_tls_ca_dir

#### Actual Results:
```
INFO - The base image option has not been specified, choosing libvirt-based test environment.
INFO - Logging into /home/jcerny/work/git/scap-security-guide/logs/rule-custom-2023-03-27-1320/test_suite.log
INFO - xccdf_org.ssgproject.content_rule_sssd_ldap_configure_tls_ca_dir
INFO - Script domain_not_there.fail.sh using profile xccdf_org.ssgproject.content_profile_stig OK
ERROR - Rule evaluation resulted in fail, instead of expected pass during final stage
ERROR - The check after remediation failed for rule 'xccdf_org.ssgproject.content_rule_sssd_ldap_configure_tls_ca_dir'.
INFO - Script ldap_tls_cacertdir.pass.sh using profile xccdf_org.ssgproject.content_profile_stig OK
INFO - Script ldap_tls_cacertdir_bad_value.fail.sh using profile xccdf_org.ssgproject.content_profile_stig OK
ERROR - Rule evaluation resulted in fail, instead of expected pass during final stage
ERROR - The check after remediation failed for rule 'xccdf_org.ssgproject.content_rule_sssd_ldap_configure_tls_ca_dir'.
INFO - Script ldap_tls_cacertdir_not_absolute_path.fail.sh using profile xccdf_org.ssgproject.content_profile_stig OK
ERROR - Rule evaluation resulted in fail, instead of expected pass during final stage
ERROR - The check after remediation failed for rule 'xccdf_org.ssgproject.content_rule_sssd_ldap_configure_tls_ca_dir'.
INFO - Script ldap_tls_cacertdir_not_there.fail.sh using profile xccdf_org.ssgproject.content_profile_stig OK
ERROR - Rule evaluation resulted in fail, instead of expected pass during final stage
ERROR - The check after remediation failed for rule 'xccdf_org.ssgproject.content_rule_sssd_ldap_configure_tls_ca_dir'.
```

#### Expected Results:
Probably no errors. But the expected results is a question that needs to be answered.

#### Additional Information/Debugging Steps:
- notice that the Playbook `logs/rule-custom-2023-03-27-1320/xccdf_org.ssgproject.content_rule_sssd_ldap_configure_tls_ca_dir.yml` doesn't contain any tasks
- notice how the `# profile` header is handled by Automatus
- notice that Automatus evaluates TS even the rule isn't a part of the STIG profile

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.