fkie-cad / fkie-cad/FACT_docker
Some plugins can't run because they can't be "installed" first.
- Dominant language
- Python
- Stars
- 29
- Forks
- 11
- PR merge metrics
- No merged PRs in 30d
Description
# Observed Issue
After following instruction from readme, FACT does run but some analysis plugins don't because they require docker images that can't be pulled. For instance, `file_system_metadata` fails with the following error:
```
[2023-09-25 13:26:16][docker][WARNING]: [file_system_metadata]: encountered docker error while processing
Process ExceptionSafeProcess-22:8:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 268, in _raise_for_status
response.raise_for_status()
File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.43/containers/create
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/docker/models/containers.py", line 819, in run
container = self.create(image=image, command=command,
File "/usr/local/lib/python3.8/dist-packages/docker/models/containers.py", line 878, in create
resp = self.client.api.create_container(**create_kwargs)
File "/usr/local/lib/python3.8/dist-packages/docker/api/container.py", line 428, in create_container
return self.create_container_from_config(config, name)
File "/usr/local/lib/python3.8/dist-packages/docker/api/container.py", line 439, in create_container_from_config
return self._result(res, True)
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 274, in _result
self._raise_for_status(response)
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 270, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.43/containers/create: Not Found ("No such image: fact/fs_metadata:latest")
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 268, in _raise_for_status
response.raise_for_status()
File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.43/images/create?tag=latest&fromImage=fact%2Ffs_metadata
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/FACT_core/src/helperFunctions/process.py", line 56, in run
raise exception
File "/opt/FACT_core/src/helperFunctions/process.py", line 51, in run
Process.run(self)
File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/FACT_core/src/analysis/PluginBase.py", line 140, in process_next_object
finished_task = self.analyze_file(task)
File "/opt/FACT_core/src/analysis/PluginBase.py", line 87, in analyze_file
fo = self.process_object(file_object)
File "/opt/FACT_core/src/plugins/analysis/file_system_metadata/code/file_system_metadata.py", line 65, in process_object
self._extract_metadata(file_object)
File "/opt/FACT_core/src/plugins/analysis/file_system_metadata/code/file_system_metadata.py", line 93, in _extract_metadata
self._extract_metadata_from_file_system(file_object)
File "/opt/FACT_core/src/plugins/analysis/file_system_metadata/code/file_system_metadata.py", line 104, in _extract_metadata_from_file_system
output = self._mount_in_docker(tmp_dir)
File "/opt/FACT_core/src/plugins/analysis/file_system_metadata/code/file_system_metadata.py", line 114, in _mount_in_docker
result = run_docker_container(
File "/opt/FACT_core/src/helperFunctions/docker.py", line 35, in run_docker_container
container = client.containers.run(image, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/docker/models/containers.py", line 822, in run
self.client.images.pull(image, platform=platform)
File "/usr/local/lib/python3.8/dist-packages/docker/models/images.py", line 444, in pull
pull_log = self.client.api.pull(
File "/usr/local/lib/python3.8/dist-packages/docker/api/image.py", line 428, in pull
self._raise_for_status(response)
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 270, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.43/images/create?tag=latest&fromImage=fact%2Ffs_metadata: Not Found ("pull access denied for fact/fs_metadata, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")
[2023-09-25 13:26:16][PluginBase][ERROR]: Worker 0: Exception during analysis file_system_metadata on 0bd402cc5ef97c260e7b1486da0f841b84ffe8c0867dd48e08083b8f6d2e1806_21528576
```
# Identified Cause
Plugins never get a chance to be _installed_ by `start.py pull`. So for the above plugin example, the required local `fact/fs_metadata` docker image is never built by the plugin code as it should.
This happens because `distribution_check(allow_unsupported=False)` is called for any plugin installer being instantiated. `distribution_check` then calls `sys.exit(1)` because `fact-core-script` docker image is based on the "unsupported" Alpine.
## In details
(code snippets below are from `fact-core-script` image):
- `start.py pull` on the host runs `/opt/FACT_core/src/install.py --backend-docker-images --frontend-docker-images` in fact-core-script container.
- In `install.py`, method `install()` has the following :
```python
only_docker = not skip_docker and none_chosen and (args.backend_docker_images or args.frontend_docker_images)
# When just pulling the docker images we don't depend on anything distribution specific
distribution = check_distribution(allow_unsupported=only_docker)
```
- Here, `skip_docker=False`, `none_chosen=True`, and both checked `args.xyz` are `True` as well.
- Therefore, `check_distribution` emits warnings about the unsupported distribution (because the script docker image uses an "unsupported" Alpine), that we can ignore (as mentioned in issue #24 among others)
- Further down it calls `install_docker_images()` which in turns call
- `backend_install_docker_images()`: runs fine
- `backend_install_plugin_docker_images()`: fails!
- The latter call, at some point: `_install_plugins()` in `install/backend.py` with
```python
_install_plugins(distribution=None, skip_docker=False, only_docker=True)
```
- This finally calls `install_docker_images()` for each plugin found in the `plugins` folder.
Now, in the above example, with `file_system_metadata` plugin, there is:
```python
class FileSystemMetadataInstaller(AbstractPluginInstaller):
base_path = Path(__file__).resolve().parent
def install_docker_images(self):
self._build_docker_image('fact/fs_metadata:latest')
```
The base class `AbstractPluginInstaller` is initialized with:
```python
def __init__(self, distribution: Optional[str] = None, skip_docker: bool = skip_docker_env):
self.distribution = distribution or check_distribution()
self.build_path = self.base_path / 'build'
self.skip_docker = skip_docker
```
Here `distribution=None` so `check_distribution()` is called with default parameters, that is `check_distribution(allow_unsupported=False)`
Then the distribution check fails with
```python
logging.critical(msg)
sys.exit(1)
```
This obviously occurs on the first plugin installer that is instantiated, and the whole installation aborts at that point.
While a human could ignore "unsupported distribution" warnings as mentioned in #24 , the installer does not, and aborts. It does not seem to be the "intended behavior" as some plugins can't run if they are not previously installed from the above.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.