internetarchive / internetarchive/draintasker

Scripts cannot be run on vanilla BusyBox distros such as Alpine

Open
#14 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
11
Forks
8
PR merge metrics
No merged PRs in 30d

Description

## Description of the issue
`pack-warcs.sh` has a dependency to GNU `findutils` package and thus cannot be run on systems or Docker containers without the `findutils` package due to the REGEX not being correctly interpreted.

## Troubleshoot and findings
Troubleshoot and findings

1. Running an Alpine docker container with correct dtmon.cfg configuration and running `draintasker/pack-warcs.sh /crawls/dtmon.cfg 1 single` with the BusyBox `find` binary give me this output :
```
job_dir = /crawls/warcs
xfer_home = /crawls/sink
warc_naming = {prefix}-{timestamp}-{serial}-{host}
item_naming = {prefix}-{timestamp14}{suffix}-{shost}
max_series_size = 10737418240 (10GB)
total_num_warcs = 0
total_size_warcs = 0
FINISH_DRAIN = /crawls/warcs/FINISH_DRAIN
OPEN = /crawls/warcs/PACKED.open
mode = single
compactify = 1
```

2. Troubleshooting it with `set -xe` shows that the computed `find` command at [pack-warcs.sh:L184](https://github.com/internetarchive/draintasker/blob/master/pack-warcs.sh#L184) gives no output :
```
[...]
++ find /crawls/warcs -maxdepth 1 -regex '.*/\(.*\)-\(.*\)-\(.*\)-\(.*\)\.w?arc\(\.gz\)?$'
+ echo ' job_dir = /crawls/warcs'
[...]
```
3. Running the same experiment on the same Docker container with `findutils` installed with `apk add findutils` gives me the following output clearly indicating that the `find` command at [pack-warcs.sh:L184](https://github.com/internetarchive/draintasker/blob/master/pack-warcs.sh#L184) is now working as intended :
```
[...]
++ find /crawls/warcs -maxdepth 1 -regex '.*/\(.*\)-\(.*\)-\(.*\)-\(.*\)\.w?arc\(\.gz\)?$'
+ for w in $(find $job_dir -maxdepth 1 -regex "${WARC_NAME_RE_FIND}")
+ (( total_num_warcs++ ))
[...]
```


## Potential solutions
I propose to either :
- rewrite parts of `pack-warcs.sh` and other impacted scripts (to be defined) to make `draintasker` POSIX-compliant
- indicate in the `README.md` that `draintasker` is intended to be run on GNU systems only
- check for the presence of GNU `findutils` at the start of `dtmon.py` and exit with an error message if it's not present
- continue troubleshooting of the different scripts and check which parts are failing on Alpine before taking a decision

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.