Climate-REF / Climate-REF/climate-ref

Ingest glob patterns with Docker/Podman/Compose

Open
#442 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
26
Forks
14
Avg merge
1d 10h
Merged PRs (30d)
36

Description

## The problem

This is likely mostly a documentation issue. The [docs currently recommend](blob/main/docs/how-to-guides/docker_deployment.md?plain=1#L45) ingesting data in a compose deployment with something like:
```bash
docker-compose run --rm climate-ref datasets ingest --source-type cmip6 /ref/data/CMIP6
```
The [README.md also recommends using globbed patterns](https://github.com/Climate-REF/climate-ref/blob/e146ffbb06dc7825150c04da35d0142dc6f397f1/README.md?plain=1#L81) for ingesting subsets of data with something like:
```bash
ref datasets ingest --source-type cmip6 path_to_archive/CMIP6/*/*/*/*/*/*mon path_to_archive/CMIP6/*/*/*/*/*/*fx --n-jobs 64
```
and states explicitly that the globs:

> should be unquoted so that they are expanded by the shell as the cli command expects a list of directories.

However, this causes problems if the container filesystem is not the same as the host where the `docker run ...` command is invoked. For example, if the archive is mounted into the container at `/ref/data`, but the path on the host is `/archive`, the host's shell won't be able to expand the globs to the correct paths.

One work around for this is to exec into the running container and run the command using the container's shell with:
```bash
docker compose run --entrypoint /bin/bash climate-ref

# Now inside the container's bash process
ref datasets ingest --source-type cmip6 /ref/data/CMIP6/*/*/*/*/*/*mon
```
but this requires the user to understand a lot about the problem.

## Definition of "done"

- The documentation should make a clear recommendation on how to handle this situation

## Additional context

I haven't looked very deeply at all into this, but python's `pathlib.Path` has `Path.glob()`, so perhaps passing the patterns directly to the cli should be a supported workflow?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.