kestra-io / kestra-io/plugin-ansible

AnsibleCLI: cache Galaxy collections and Python dependencies across executions

Open
#135 0 comments 0 reactions 0 assignees View on GitHub
area/plugin kind/customer-request kind/performance
Dominant language
Java
Stars
2
Forks
6
Avg merge
17h 5m
Merged PRs (30d)
6

Description

## Summary

`AnsibleCLI` reinstalls Galaxy collections and Python packages on every task execution, with nothing carried over between runs. Give it first-class dependency caching, mirroring `dependencies` + `dependencyCacheEnabled` on the Python script tasks.

## Current behaviour

Since #74, the plugin auto-installs from files found in the working directory:

- `autoInstallGalaxyRequirements` (default `true`) runs `ansible-galaxy install -r requirements.yml`
- `autoInstallPythonRequirements` (default `true`) runs `pip install --no-cache-dir -r requirements.txt`

Both re-run in full on every execution. `--no-cache-dir` also disables pip's own HTTP cache, so nothing is reused even within the same worker. `ansible-galaxy` re-downloads collections into the working directory each time. `beforeCommands` has the same characteristic: it runs once per task execution and is not reused across the multiple ansible commands in a task, let alone across executions.

## Use case

Users declare a fixed set of Galaxy collections (plus the Python libs some of them need, e.g. `proxmoxer` for `community.proxmox`) so their playbooks can use them. The collection set changes rarely, but the install cost is paid on every single execution, which is the dominant part of runtime for short playbooks.

Two ways users work around this today:

- Pre-bake the collections into a custom `containerImage`. Works well, but each change needs an image build and, in regulated environments, an internal approval cycle. That approval cost is what makes this a non-starter for some users.
- Wrap `AnsibleCLI` in a `WorkingDirectory` task, have `beforeCommands` install into a subdirectory, and cache that subdirectory with a TTL ([caching docs](https://kestra.io/docs/concepts/caching)). This uses existing, supported features, but it requires restructuring the flow and hand-managing the cache path and TTL for something that is really a plugin-level concern.

## Proposal

Give Ansible the same treatment Python tasks already have:

- `dependencies` on `AnsibleCLI`: a list of collection/role specifiers (equivalent to the entries of a `requirements.yml`), so the set can be declared on the task instead of shipped as a file.
- `dependencyCacheEnabled` (default `true`): cache the resolved collections and their Python dependencies across executions, keyed on the resolved dependency set plus the container image, the way `io.kestra.plugin.scripts.python.Commands` already does.

Ideally the existing `autoInstall*` paths hit the same cache when `requirements.yml` / `requirements.txt` arrive via `namespaceFiles` or `inputFiles`, since that is how most users declare them today. Worth deciding whether the cache stores the installed collection tree or just the downloaded artifacts, given that some collections build native Python deps.

## Current Workaround

`WorkingDirectory` + `cache` with a TTL, or pre-baking into a custom container image. Both work; neither is first-class.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the AnsibleCLI entry point and compare its dependency installation paths with the Python script tasks' existing dependencies and dependencyCacheEnabled behavior. Review how autoInstallGalaxyRequirements and autoInstallPythonRequirements handle requirements.yml and requirements.txt. Done should include declared dependencies and reuse across executions, with the cache key and installed-versus-downloaded contents resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
ansible, python
Domain
devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.