internetarchive / internetarchive/heritrix3

Docker

Open
#360 9 comments 2 reactions 0 assignees View on GitHub
feature request pull request welcome
Dominant language
Java
Stars
3.3k
Forks
793
Avg merge
1d 8h
Merged PRs (30d)
8

Description

I wrote a Docker file for the current version(s). Maybe you want to look into it and integrate it here.
It works for me but I only have some simple use-cases (like API tests with python3), so I do not know how it performs under stress. And whether users require more configuration options. (But they could theoretically bind-mount other files if required.)

See Docker-Hub: https://hub.docker.com/r/ekoerner/heritrix

My `Dockerfile` (currently in private repository, so I can't provide any link, just the content here)
```Dockerfile
ARG java=11-jre

FROM openjdk:${java}

ARG version="3.4.0-20210923"
ARG contrib=0
ARG user="heritrix"
ARG userid=1000

LABEL version=${version}
LABEL contrib=${contrib}
LABEL user=${user}/$userid

# create user
RUN \
groupadd -g $userid $user && \
useradd -r -u $userid -g $user $user

# install other requirements (for contrib)
RUN \
if [ ${contrib} -eq 1 ] ; then \
apt-get update && \
apt-get install -y --no-install-recommends \
youtube-dl && \
rm -rf /var/lib/apt/lists/* ; \
fi

WORKDIR /opt

# download latest version according to:
# https://github.com/internetarchive/heritrix3/releases/tag/3.4.0-20210923
RUN \
if [ ${contrib} -eq 1 ] ; then \
wget -O heritrix-contrib-${version}-dist.tar.gz https://repo1.maven.org/maven2/org/archive/heritrix/heritrix-contrib/${version}/heritrix-contrib-${version}-dist.tar.gz && \
tar xvfz heritrix-contrib-${version}-dist.tar.gz && \
rm heritrix-contrib-${version}-dist.tar.gz && \
mv heritrix-contrib-${version} heritrix ; \
else \
wget -O heritrix-${version}-dist.zip https://repo1.maven.org/maven2/org/archive/heritrix/heritrix/${version}/heritrix-${version}-dist.zip && \
unzip heritrix-${version}-dist.zip && \
rm heritrix-${version}-dist.zip && \
mv heritrix-${version} heritrix ; \
fi && \
chmod u+x heritrix/bin/heritrix && \
chown -R $user:$user /opt/heritrix

# create a run script because dynamic configuration of credentials
RUN printf '%s\n' \
'#!/bin/bash' \
'' \
'_JOBARGS="-b /"' \
'' \
'# set credentials (require both USERNAME and PASSWORD)' \
'# -a "${USERNAME}:${PASSWORD}"' \
'if [[ ! -z "$USERNAME" ]] && [[ ! -z "$PASSWORD" ]]; then' \
' echo "${USERNAME}:${PASSWORD}" > ${HERITRIX_HOME}/credentials.txt' \
' _JOBARGS="$_JOBARGS -a @${HERITRIX_HOME}/credentials.txt"' \
'elif [[ ! -z "$CREDSFILE" ]]; then' \
' _JOBARGS="$_JOBARGS -a @${CREDSFILE}"' \
'else' \
' >&2 echo "No USERNAME and/or PASSWORD environment var set!"' \
'fi' \
'' \
'# check if -r mode' \
'if [[ ! -z "$JOBNAME" ]]; then' \
' >&2 echo "Found JOBNAME envvar, just running job: $JOBNAME"' \
' _JOBARGS="$_JOBARGS -r $JOBNAME"' \
' if [ ! -f "/opt/heritrix/jobs/$JOBNAME/crawler-beans.cxml" ]; then' \
' >&2 echo "Did not find any '"'"'crawler-beans.cxml'"'"' for job '"'"'$JOBNAME'"'"'!"' \
' fi' \
'fi' \
'' \
'# run' \
'exec ${HERITRIX_HOME}/bin/heritrix $_JOBARGS' \
'' \
> heritrix.sh && \
chmod +x heritrix.sh && \
chown $user:$user heritrix.sh

WORKDIR /opt/heritrix

USER $user

ENV HERITRIX_HOME /opt/heritrix
# let it run in the foreground, required for docker
ENV FOREGROUND true

# standard webport
# NOTE: that the webpage is via HTTPS only available!
EXPOSE 8443

CMD ["/opt/heritrix.sh"]
```

Build it:
```bash
docker build --build-arg version=3.4.0-20210923 -t heritrix .
```
Build `heritrix-contrib` (requires Java 8, with Java 11 (JRE/JDK) some JNI error, maybe related to #265?)
```bash
docker build --build-arg version=3.4.0-20210923 --build-arg contrib=1 --build-arg java=8-jre -t heritrix-contrib .
```

Example `docker-compose.yml` (also on DockerHub currently)
```yml
version: "3.7"
services:

heritrix:
build: .
container_name: "heritrix"
# TEST: keeps the container running without doing anything (for inspections)
# entrypoint: bash -c 'while :; do :; done & kill -STOP $$! && wait $$!'
# env_file: .env
environment:
- USERNAME=admin
- PASSWORD=admin
# optional jobname to run (will only run this single job and exit!)
# - JOBNAME=myjob
# - JAVA_OPTS=-Xmx1024M
init: true
ports:
# if you want to use a .env file with `PORT=8443` for example
# - ${PORT}:8443
- 8443:8443
restart: unless-stopped
volumes:
# where jobs will be stored
- job-files:/opt/heritrix/jobs
# or if JOBNAME envvar is used (mount just the single job folder)
# jobfolder in the container needs to have the same name as in JOBNAME
# - $(pwd)/host_myjob:/opt/heritrix/jobs/myjob

volumes:
job-files:
```

---

**UPDATE:** I added the `-r ` option to my image on dockerhub. Simply set the `JOBNAME=jobname` environment variable to run the job `jobname`. Take care to mount the (preconfigured) job folder into the image, see above. Only works from version *3.4.0-20210803*, see pull request #406.
**UPDATE2:** I added a `contrib` image that uses `heritrix-contrib`. For now it only includes `youtube-dl` as extra dependency and it only works with Java 8 JRE. The `contrib` image is only available from version *3.4.0-20210923*.
**UPDATE3:** Added a custom user to make it a bit more secure (e. g., no package installs possible anymore). Note that `-b /` is required to make the web UI visible in the docker image.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.