internetarchive / internetarchive/heritrix3
Docker
- Dominant language
- Java
- Stars
- 3.3k
- Forks
- 793
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 8
Description
I wrote a Docker file for the current version(s). Maybe you want to look into it and integrate it here.
It works for me but I only have some simple use-cases (like API tests with python3), so I do not know how it performs under stress. And whether users require more configuration options. (But they could theoretically bind-mount other files if required.)
See Docker-Hub: https://hub.docker.com/r/ekoerner/heritrix
My `Dockerfile` (currently in private repository, so I can't provide any link, just the content here)
```Dockerfile
ARG java=11-jre
FROM openjdk:${java}
ARG version="3.4.0-20210923"
ARG contrib=0
ARG user="heritrix"
ARG userid=1000
LABEL version=${version}
LABEL contrib=${contrib}
LABEL user=${user}/$userid
# create user
RUN \
groupadd -g $userid $user && \
useradd -r -u $userid -g $user $user
# install other requirements (for contrib)
RUN \
if [ ${contrib} -eq 1 ] ; then \
apt-get update && \
apt-get install -y --no-install-recommends \
youtube-dl && \
rm -rf /var/lib/apt/lists/* ; \
fi
WORKDIR /opt
# download latest version according to:
# https://github.com/internetarchive/heritrix3/releases/tag/3.4.0-20210923
RUN \
if [ ${contrib} -eq 1 ] ; then \
wget -O heritrix-contrib-${version}-dist.tar.gz https://repo1.maven.org/maven2/org/archive/heritrix/heritrix-contrib/${version}/heritrix-contrib-${version}-dist.tar.gz && \
tar xvfz heritrix-contrib-${version}-dist.tar.gz && \
rm heritrix-contrib-${version}-dist.tar.gz && \
mv heritrix-contrib-${version} heritrix ; \
else \
wget -O heritrix-${version}-dist.zip https://repo1.maven.org/maven2/org/archive/heritrix/heritrix/${version}/heritrix-${version}-dist.zip && \
unzip heritrix-${version}-dist.zip && \
rm heritrix-${version}-dist.zip && \
mv heritrix-${version} heritrix ; \
fi && \
chmod u+x heritrix/bin/heritrix && \
chown -R $user:$user /opt/heritrix
# create a run script because dynamic configuration of credentials
RUN printf '%s\n' \
'#!/bin/bash' \
'' \
'_JOBARGS="-b /"' \
'' \
'# set credentials (require both USERNAME and PASSWORD)' \
'# -a "${USERNAME}:${PASSWORD}"' \
'if [[ ! -z "$USERNAME" ]] && [[ ! -z "$PASSWORD" ]]; then' \
' echo "${USERNAME}:${PASSWORD}" > ${HERITRIX_HOME}/credentials.txt' \
' _JOBARGS="$_JOBARGS -a @${HERITRIX_HOME}/credentials.txt"' \
'elif [[ ! -z "$CREDSFILE" ]]; then' \
' _JOBARGS="$_JOBARGS -a @${CREDSFILE}"' \
'else' \
' >&2 echo "No USERNAME and/or PASSWORD environment var set!"' \
'fi' \
'' \
'# check if -r mode' \
'if [[ ! -z "$JOBNAME" ]]; then' \
' >&2 echo "Found JOBNAME envvar, just running job: $JOBNAME"' \
' _JOBARGS="$_JOBARGS -r $JOBNAME"' \
' if [ ! -f "/opt/heritrix/jobs/$JOBNAME/crawler-beans.cxml" ]; then' \
' >&2 echo "Did not find any '"'"'crawler-beans.cxml'"'"' for job '"'"'$JOBNAME'"'"'!"' \
' fi' \
'fi' \
'' \
'# run' \
'exec ${HERITRIX_HOME}/bin/heritrix $_JOBARGS' \
'' \
> heritrix.sh && \
chmod +x heritrix.sh && \
chown $user:$user heritrix.sh
WORKDIR /opt/heritrix
USER $user
ENV HERITRIX_HOME /opt/heritrix
# let it run in the foreground, required for docker
ENV FOREGROUND true
# standard webport
# NOTE: that the webpage is via HTTPS only available!
EXPOSE 8443
CMD ["/opt/heritrix.sh"]
```
Build it:
```bash
docker build --build-arg version=3.4.0-20210923 -t heritrix .
```
Build `heritrix-contrib` (requires Java 8, with Java 11 (JRE/JDK) some JNI error, maybe related to #265?)
```bash
docker build --build-arg version=3.4.0-20210923 --build-arg contrib=1 --build-arg java=8-jre -t heritrix-contrib .
```
Example `docker-compose.yml` (also on DockerHub currently)
```yml
version: "3.7"
services:
heritrix:
build: .
container_name: "heritrix"
# TEST: keeps the container running without doing anything (for inspections)
# entrypoint: bash -c 'while :; do :; done & kill -STOP $$! && wait $$!'
# env_file: .env
environment:
- USERNAME=admin
- PASSWORD=admin
# optional jobname to run (will only run this single job and exit!)
# - JOBNAME=myjob
# - JAVA_OPTS=-Xmx1024M
init: true
ports:
# if you want to use a .env file with `PORT=8443` for example
# - ${PORT}:8443
- 8443:8443
restart: unless-stopped
volumes:
# where jobs will be stored
- job-files:/opt/heritrix/jobs
# or if JOBNAME envvar is used (mount just the single job folder)
# jobfolder in the container needs to have the same name as in JOBNAME
# - $(pwd)/host_myjob:/opt/heritrix/jobs/myjob
volumes:
job-files:
```
---
**UPDATE:** I added the `-r ` option to my image on dockerhub. Simply set the `JOBNAME=jobname` environment variable to run the job `jobname`. Take care to mount the (preconfigured) job folder into the image, see above. Only works from version *3.4.0-20210803*, see pull request #406.
**UPDATE2:** I added a `contrib` image that uses `heritrix-contrib`. For now it only includes `youtube-dl` as extra dependency and it only works with Java 8 JRE. The `contrib` image is only available from version *3.4.0-20210923*.
**UPDATE3:** Added a custom user to make it a bit more secure (e. g., no package installs possible anymore). Note that `-b /` is required to make the web UI visible in the docker image.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.