ComplianceAsCode / ComplianceAsCode/content

"<" operator in jq filter fails the build

Open
#11,922 4 comments 0 reactions 1 assignee Claimed by @yuumasato View on GitHub
triaged
Dominant language
Shell
Stars
2.8k
Forks
828
Avg merge
3d 8m
Merged PRs (30d)
80

Description

#### Description of problem:
I tried to use the less-than-operator "<" in jq filter which should be supported according to https://jqlang.github.io/jq/
Everytime I try to run a cluster-test with using the corresponding rule the build fails:

```
* Testing rule kube_descheduler_lifecycle_policy in-cluster
* Pushing image build to cluster
2024-04-30 11:33:22,749:INFO: Building content for ocp4, rhcos4
namespace/openshift-compliance unchanged
2024-04-30 11:33:23,703:DEBUG: Created namespace openshift-compliance
-- SCAP Security Guide 0.1.73
-- (see /home/mermer/content/docs/manual/developer_guide.adoc for build instructions)
--
-- Found PythonInterp: /usr/bin/python3 (found version "3.9.18")
-- Found PY_yaml: /usr/lib64/python3.9/site-packages/yaml
-- Found PY_jinja2: /usr/lib/python3.9/site-packages/jinja2
-- Found PY_lxml: /usr/lib64/python3.9/site-packages/lxml
-- Could NOT find PY_pytest (missing: PY_PYTEST)
-- Could NOT find PY_pytest_cov (missing: PY_PYTEST_COV)
-- Could NOT find PY_json2html (missing: PY_JSON2HTML)
-- Could NOT find PY_mypy (missing: PY_MYPY)
-- Could NOT find PY_openpyxl (missing: PY_OPENPYXL)
-- Could NOT find PY_pandas (missing: PY_PANDAS)
-- Could NOT find PY_pcre2 (missing: PY_PCRE2)
-- Could NOT find PY_cmakelint (missing: PY_CMAKELINT)
-- Could NOT find PY_github (missing: PY_GITHUB)
-- Could NOT find PY_sphinx (missing: PY_SPHINX)
-- Could NOT find PY_sphinxcontrib.autojinja (missing: PY_SPHINXCONTRIB.AUTOJINJA)
-- Could NOT find PY_sphinx_rtd_theme (missing: PY_SPHINX_RTD_THEME)
-- Could NOT find PY_myst_parser (missing: PY_MYST_PARSER)
-- Could NOT find PY_prometheus_client (missing: PY_PROMETHEUS_CLIENT)
-- Could NOT find PY_trestle (missing: PY_TRESTLE)
-- Found PY_requests: /usr/lib/python3.9/site-packages/requests
-- CMake:
-- build type: Debug
-- generator: Unix Makefiles
-- source directory: /home/mermer/content
-- build directory: /home/mermer/content/build
-- Logging: OFF
--
-- Tools:
-- python: /usr/bin/python3 (version: 3.9.18)
-- python yaml module: /usr/lib64/python3.9/site-packages/yaml
-- python jinja2 module: /usr/lib/python3.9/site-packages/jinja2
-- oscap: /usr/bin/oscap (version: 1.3.8)
-- xsltproc: /usr/bin/xsltproc
-- xmllint: /usr/bin/xmllint
-- sed: /usr/bin/sed
-- shellcheck (optional): SHELLCHECK_EXECUTABLE-NOTFOUND
-- linkchecker (optional): LINKCHECKER_EXECUTABLE-NOTFOUND
-- grep (optional): /usr/bin/grep
-- python pytest module (optional):
-- ansible-playbook module (optional): /usr/bin/ansible-playbook
-- ansible-lint module (optional): /usr/bin/ansible-lint
-- yamllint module (optional): YAMLLINT_EXECUTABLE-NOTFOUND
-- python mypy module (optional):
-- BATS framework (optional): BATS_EXECUTABLE-NOTFOUND
-- python sphinx module (optional):
-- python sphinxcontrib.autojinja module (optional):
-- python sphinx_rtd_theme module (optional):
-- python myst-parser module (optional):
-- python openpyxl module (optional):
-- python pandas module (optional):
-- python pcre2 module (optional):
-- python lxml module (optional): /usr/lib64/python3.9/site-packages/lxml
-- python prometheus-client module (optional):
-- python compliance-trestle module (optional):
-- python github (PyGitHub) module (optional):
--
-- Build options:
-- SSG vendor string: ssgproject
-- Target OVAL version: 5.11
-- Build SCAP 1.2 source data streams: ON
-- OVAL schematron validation: ON
-- shellcheck bash fixes validation: ON
-- Separate SCAP files: ON
-- Ansible Playbooks: ON
-- Ansible Playbooks Per Rule: OFF
-- Bash scripts: ON
-- Thin data streams: OFF
-- jinja2 cache: enabled
-- jinja2 cache dir: /home/mermer/content/build/jinja2_cache
-- STIG Delta Taloring files: ON
-- Build SCE Content: OFF
-- SCAPVal 1.3 Enabled: OFF
--
-- Products:
-- Alibaba Cloud Linux 2: OFF
-- Alibaba Cloud Linux 3: OFF
-- Anolis OS 8: OFF
-- Anolis OS 23: OFF
-- Chromium: OFF
-- Debian 10: OFF
-- Debian 11: OFF
-- Debian 12: OFF
-- Example: OFF
-- EKS: OFF
-- Fedora: OFF
-- Firefox: OFF
-- MacOS 1015: OFF
-- OCP4: ON
-- RHCOS4: ON
-- Oracle Linux 7: OFF
-- Oracle Linux 8: OFF
-- Oracle Linux 9: OFF
-- openEuler 22.03 LTS: OFF
-- openSUSE: OFF
-- RHEL 7: OFF
-- RHEL 8: OFF
-- RHEL 9: OFF
-- RHEL 10: OFF
-- RHV 4: OFF
-- SUSE 12: OFF
-- SUSE 15: OFF
-- Ubuntu 16.04: OFF
-- Ubuntu 18.04: OFF
-- Ubuntu 20.04: OFF
-- Ubuntu 22.04: OFF
-- Uos 20: OFF
-- OpenEmbedded: OFF
--
-- Scanning for dependencies of ocp4 fixes (bash, ansible, puppet, anaconda, ignition, kubernetes and blueprint)...
-- Won't validate ocp4, as it requires the OpenSCAP scanner that is capable of the validation.
-- Won't validate ocp4, as it requires the OpenSCAP scanner that is capable of the validation.
-- Won't validate ocp4, as it requires the OpenSCAP scanner that is capable of the validation.
-- Scanning for dependencies of rhcos4 fixes (bash, ansible, puppet, anaconda, ignition, kubernetes and blueprint)...
-- Configuring done
-- Generating done
-- Build files have been written to: /home/mermer/content/build
[ 0%] [rhcos4-content] generating sce/metadata.json
[ 0%] Built target ocp4-tables
[ 0%] [ocp4-content] generating sce/metadata.json
[ 0%] Built target generate-internal-rhcos4-sce-metadata.json
[ 0%] Built target generate-internal-ocp4-sce-metadata.json
[ 0%] [rhcos4-content] compiling product yaml
[ 0%] [ocp4-content] compiling product yaml
[ 0%] [rhcos4-content] compiling everything
[ 0%] [ocp4-content] compiling everything
[ 1%] Built target ocp4-compile-all
[ 1%] [ocp4-content] generating templated content
[ 1%] Generating ../tables/tables-ocp4-all.html
[ 2%] Built target generate-ssg-tables-ocp4-all
[ 4%] Built target generate-internal-templated-content-ocp4
[ 4%] [ocp4-content] generating cpe-oval-unlinked.xml
[ 4%] [ocp4-content] generating oval-unlinked.xml
[ 4%] [ocp4-content] collecting all fixes
[ 5%] Built target generate-internal-ocp4-cpe-oval-unlinked.xml
[ 7%] Built target generate-internal-ocp4-all-fixes
[ 10%] Built target generate-internal-ocp4-oval-unlinked.xml
[ 10%] [ocp4-content] generating plain XCCDF, OVAL and OCIL files
Traceback (most recent call last):
File "/home/mermer/content/ssg/entities/common.py", line 111, in add_sub_element
element = ET.fromstring(ustr.encode("utf-8"))
File "/usr/lib64/python3.9/xml/etree/ElementTree.py", line 1342, in XML
parser.feed(text)
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 14, column 152

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/mermer/content/build-scripts/build_xccdf.py", line 148, in
main()
File "/home/mermer/content/build-scripts/build_xccdf.py", line 127, in main
xccdftree = loader.export_benchmark_to_xml()
File "/home/mermer/content/ssg/build_yaml.py", line 1689, in export_benchmark_to_xml
_, benchmark = self.benchmark.get_benchmark_xml_for_profiles(
File "/home/mermer/content/ssg/build_yaml.py", line 556, in get_benchmark_xml_for_profiles
return profiles_ids, self.to_xml_element(
File "/home/mermer/content/ssg/build_yaml.py", line 504, in to_xml_element
self._add_groups_xml(root, components_to_not_include, env_yaml)
File "/home/mermer/content/ssg/build_yaml.py", line 471, in _add_groups_xml
root.append(group.to_xml_element(env_yaml, components_to_not_include))
File "/home/mermer/content/ssg/build_yaml.py", line 759, in to_xml_element
self._add_sub_groups(group, components_to_not_include, env_yaml)
File "/home/mermer/content/ssg/build_yaml.py", line 732, in _add_sub_groups
group.append(_group.to_xml_element(env_yaml, components_to_not_include))
File "/home/mermer/content/ssg/build_yaml.py", line 758, in to_xml_element
self._add_rules_xml(group, rules_to_not_include, env_yaml)
File "/home/mermer/content/ssg/build_yaml.py", line 687, in _add_rules_xml
group.append(rule.to_xml_element(env_yaml))
File "/home/mermer/content/ssg/build_yaml.py", line 1185, in to_xml_element
add_warning_elements(rule, self.warnings)
File "/home/mermer/content/ssg/build_yaml.py", line 91, in add_warning_elements
warning = add_sub_element(
File "/home/mermer/content/ssg/entities/common.py", line 115, in add_sub_element
raise RuntimeError(msg)
RuntimeError: Error adding subelement to an element '{http://checklists.nist.gov/xccdf/1.2}Rule' from string: 'This rule's check operates on the cluster configuration dump.
Therefore, you need to use a tool that can query the OCP API, retrieve the following:








/apis/operator.openshift.io/v1/namespaces/openshift-kube-descheduler-operator/kubedeschedulers/cluster
API endpoint, filter with with the jq utility using the following filter
[.spec.deschedulingIntervalSeconds <= 3600]
and persist it to the local
/apis/operator.openshift.io/v1/namespaces/openshift-kube-descheduler-operator/kubedeschedulers/cluster#3915f675b1f452d2e8d5cb4ad2788527eb758e902db4c8423d7159572a3943bb
file.

'
make[2]: *** [ocp4/CMakeFiles/generate-ocp4-xccdf-oval-ocil.dir/build.make:77: ocp4/ssg-ocp4-xccdf.xml] Error 1
make[1]: *** [CMakeFiles/Makefile2:682: ocp4/CMakeFiles/generate-ocp4-xccdf-oval-ocil.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 10%] Built target rhcos4-compile-all
make: *** [Makefile:154: all] Error 2
Traceback (most recent call last):
File "/home/mermer/content/utils/build_ds_container.py", line 305, in
build_content_locally(args.products)
File "/home/mermer/content/utils/build_ds_container.py", line 125, in build_content_locally
subprocess.run(command, check=True, capture_output=CAPTURE_OUTPUT)
File "/usr/lib64/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/mermer/content/build_product', 'ocp4', 'rhcos4', '--debug']' returned non-zero exit status 2.
```

Using other comparison operators work as expected.
The issue could be because of the conversion to XML and interpreting "<" as XML syntax.
I tried escaping the character without success.

#### SCAP Security Guide Version:
0.1.73

#### Operating System Version:
RHEL 9.3

#### Steps to Reproduce:

The rule I tried to create is specific to the kube descheduler operator so I'll describe how to reproduce for this specific issue.

1. Create a cluster role and rolebinding for the compliance operator to have permissions for openshift-kube-descheduler-operator:kubedeschedulers

2. Write rule as follows:
```
documentation_complete: true

title: Ensure that the LifecycleAndUtilization profile for the Kube Descheduler operator is enabled

description: |-
TBD

rationale: |-
TBD

identifiers: {}

references:
bsi: APP.4.4.A21

severity: medium

ocil_clause: "TBD"

ocil: |-
TBD

{{% set jqfilter = '[.spec.deschedulingIntervalSeconds < 3600]' %}}

warnings:
- general: |-
{{{ openshift_filtered_cluster_setting({'/apis/operator.openshift.io/v1/namespaces/openshift-kube-descheduler-operator/kubedeschedulers/cluster': jqfilter}) | indent(4) }}}

template:
name: yamlfile_value
vars:
ocp_data: "true"
filepath: {{{ openshift_filtered_path('/apis/operator.openshift.io/v1/namespaces/openshift-kube-descheduler-operator/kubedeschedulers/cluster', jqfilter) }}}
yamlpath: "[:]"
check_existence: "all_exist"
entity_check: "all"
values:
- value: "true"
operation: "equals"
```

3. Start a cluster-test using this rule
`./utils/add_kubernetes_rule.py cluster-test --rule kube_descheduler_lifecycle_policy --skip-deploy`

#### Actual Results:

Build fails

#### Expected Results:

Build and cluster-test finishes

#### Additional Information/Debugging Steps:

The issue can probably be reproduced not only with the kube-descheduler operator as long as the "<"-operator is used in the jqfilter.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.