ComplianceAsCode / ComplianceAsCode/content

Automatus tests fail when rules marked with 'platform: machine' are run in a container environment

Open
#12,511 0 comments 0 reactions 0 assignees View on GitHub
triaged
Dominant language
Shell
Stars
2.8k
Forks
828
Avg merge
3d 8m
Merged PRs (30d)
80

Description

#### Description of problem:

Automatus tests fail with "Environment failed to prepare" when rules, which are not applicable to containers (e.g. platform is machine,system_with_kernel,systemd,...), are tested in a container environment, and which try to manipulate a component which is not applicable to containers (kernel modules, sysctl, systemd services, apparmor profiles, ...).

An example rule which tests fine in a VM but fails to test in a container is `all_apparmor_profiles_in_enforce_complain_mode`

```
$ tests/test_rule_in_container.sh --no-make-applicable-in-containers --dontclean --logdir logs_bash --remediate-using bash --name ssg_test_suite --datastream build/ssg-ubuntu2204-ds.xml all_apparmor_profiles_in_enforce_complain_mode

ERROR - Rule 'all_apparmor_profiles_in_enforce_complain_mode' test setup script 'correct_all_apparmor_profiles_in_enforce.pass.sh' failed with exit code 1
ERROR - Environment failed to prepare, skipping test
```

By adding the following to the test scenario, the issue is resolved:
```
if [ -f /.dockerenv ] || [ -f /run/.containerenv ]; then
exit 0
fi
```

```
$ tests/test_rule_in_container.sh --no-make-applicable-in-containers --dontclean --logdir logs_bash --remediate-using bash --name ssg_test_suite --datastream build/ssg-ubuntu2204-ds.xml all_apparmor_profiles_in_enforce_complain_mode

WARNING - Script correct_all_apparmor_profiles_in_enforce.pass.sh using profile (all) notapplicable
WARNING - Rule xccdf_org.ssgproject.content_rule_all_apparmor_profiles_in_enforce_complain_mode evaluation resulted in notapplicable
```

#### Reason:

The ssg test suite does not take into account the various applicability definitions in the rule, when deciding which tests to run. The evaluation of applicability (machine vs container) is performed only after the test environment is already prepared, which can result in the test failing if the environment is not setup correctly in a non-supported environment.

#### SCAP Security Guide Version:
master branch (https://github.com/ComplianceAsCode/content/commit/a11711d14d3252adc05916d5a9419eabd9cce10c)

#### Operating System Version:

#### Steps to Reproduce:

1. Add `platform: machine` to rule `all_apparmor_profiles_in_enforce_complain_mode`
2. `./build_product ubuntu2204`
3. `podman build --build-arg "CLIENT_PUBLIC_KEY=$(cat ~/.ssh/id_rsa.pub)" -t ssg_test_suite -f Dockefiles/test_suite-ubuntu2204`
4. `tests/test_rule_in_container.sh --no-make-applicable-in-containers --dontclean --logdir logs_bash --remediate-using bash --name ssg_test_suite --datastream build/ssg-ubuntu2204-ds.xml all_apparmor_profiles_in_enforce_complain_mode`

#### Actual Results:
See error above

#### Expected Results:
The expected result is that the tests pass when inside a container because the rule should not be evaluated.

#### Additional Information/Debugging Steps:
Possible solutions that come to mind:
- the tests should be removed from the set of applicable tests in a container environment
- scenarios are appended with the conditional to return 0 if running in a container environment

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.