get2knowio / get2knowio/remo

Move boto3 to an optional `aws` extra and scope the Ansible collection install per provider

Open
#94 0 comments 0 reactions 0 assignees View on GitHub
dependencies enhancement tech-debt
Dominant language
Python
Stars
2
Forks
0
Avg merge
15h 59m
Merged PRs (30d)
9

Description

## Summary

Move `boto3` out of the unconditional runtime dependencies into an optional `aws` extra, and scope the
Ansible collection install to the provider actually being used. Today every user — including someone who
only ever touches Incus or Proxmox — installs the full AWS SDK and the AWS Ansible collections.

Split out of #019 (dependency/dead-code/documentation hygiene), which deliberately limited itself to
making the *existing* dependency declarations truthful. This issue is the behavioral follow-up.

## Measurements

Clean `uv venv` installs, Python 3.12, no extras. Default install is **65 MB** of `site-packages`:

| Stack | Size | % of install |
|---|---:|---:|
| **boto3** (botocore 24.6, boto3 1.05, urllib3 0.49, dateutil 0.47, s3transfer 0.35, jmespath 0.08, six 0.04) | **27.1 MB** | **42%** |
| **hcloud** (hcloud 0.70, idna 0.34, requests 0.26, certifi 0.25, charset_normalizer 0.23) | **1.8 MB** | **2.7%** |
| Combined | **28.9 MB** | **45%** |

`botocore` alone is 24.6 MB — **38% of the entire install**. Verified end-to-end: `click + InquirerPy +
ansible-core` alone is 34 MB versus 65 MB for the full set (31 MB delta, 29 MB of it dependencies and
1.3 MB the `remo_cli` package itself).

What each user would actually save:

| User | Saves | % of venv |
|---|---:|---:|
| Incus / Proxmox only | 28.9 MB | 45% |
| Hetzner only (drops boto3) | 27.1 MB | 42% |
| AWS only (drops hcloud) | 1.8 MB | 2.7% |

**`boto3` is 94% of the available savings; `hcloud` is 6%.** Hence this issue targets `boto3` only —
making `hcloud` optional buys 2.7% while introducing an install-time failure mode (see below).

## The collection layer has to move with it

`core/ansible_runner.py::_ensure_collections()` runs `ansible-galaxy collection install -r
requirements.yml` — **all** collections, unconditionally, on the first playbook run of *any* provider.
That is another **41 MB** in `~/.ansible/collections`:

```
community.general 20 MB (needed by everyone)
amazon.aws 9 MB <- AWS only
community.aws 6 MB <- AWS only
community.crypto 4 MB
hetzner.hcloud 3 MB <- Hetzner only
ansible.posix 2 MB
```

So an Incus-only user's real footprint is ~106 MB, of which ~44 MB is provider-specific — and only
29 MB of that is reachable through pyproject extras. **Doing the extra without scoping the collection
install is a half-measure that also creates a new failure mode**: `amazon.aws` would install fine, then
its modules would die at runtime for want of `boto3`.

## Existing preflight gap (must be closed as part of this)

`ansible/roles/hetzner_server/tasks/main.yml:10-26` checks for `hcloud` and pip-installs it if missing.
The equivalent guard does **not** exist for AWS anywhere, nor for Hetzner teardown/resize:

- `ansible/aws_teardown.yml`, `ansible/aws_resize.yml`, `ansible/roles/aws_server/tasks/*` — no boto3 preflight
- `ansible/hetzner_teardown.yml`, `ansible/hetzner_resize.yml` — no hcloud preflight

Both `ansible/hetzner_site.yml:10` and `ansible/hetzner_teardown.yml:14` pin
`ansible_python_interpreter: "{{ ansible_playbook_python }}"` with the comment *"Use the same Python that
runs ansible-playbook (has hcloud installed)"* — i.e. the collections rely on the CLI's own environment
carrying the SDK. That assumption is exactly what this change breaks, deliberately, so the guards have to
be added first.

## Proposed scope

- [ ] Add an `aws` optional extra carrying `boto3`; remove it from unconditional `dependencies`.
- [ ] Point `aws_descriptor.sdk_extra` at the now-real `aws` extra (019 makes this field truthful; this
issue makes it load-bearing again).
- [ ] Restore the lazy-import guard in `providers/aws.py` to a reachable state — `_require_boto3()`
currently guards a condition that cannot occur.
- [ ] Scope `_ensure_collections()` to install only the collections the invoked provider needs, keyed off
the provider descriptor rather than a single global requirements hash. The `collections.lock`
marker becomes per-scope.
- [ ] Add a boto3-present preflight to the AWS playbooks, matching the `hetzner_server` role pattern
(check → `pip install` → proceed), and backfill the same for Hetzner teardown/resize.
- [ ] Update install documentation for the new extra, and the smoke workflows that assume a single
`uv sync --all-extras` environment.

## Explicitly not in scope

Making `hcloud` optional. At 1.8 MB (2.7%) it does not justify a second install-time failure mode. It
stays a hard dependency, annotated with its `hetzner.hcloud` consumer per #019.

## Acceptance

- A default `pip install remo-cli` produces a `site-packages` tree ~27 MB smaller than today.
- `remo aws create` on a default install fails with the existing actionable "install the AWS extra"
message rather than a traceback or an Ansible module error.
- `remo aws create` with the `aws` extra installed succeeds, including when `destroy` or `resize` is the
first AWS command ever run.
- An Incus-only user never has `amazon.aws` or `community.aws` installed.

Contributor guide

Open the contributing guide

Research direction

Start with the dependency declarations, core/ansible_runner.py::_ensure_collections(), providers/aws.py, and the AWS and Hetzner playbooks and role paths named in the issue. Trace provider descriptors, collection markers, and the existing Hetzner preflight pattern, then run the smoke workflows. Done means the default install omits boto3 and AWS collections, AWS extra commands work including destroy and resize, and Incus-only runs do not install AWS collections.

Written by the indexing model from the issue text.

Assessment

Tech stack
ansible, aws, python
Domain
build-system, cloud, devops
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.