kestra-io / kestra-io/plugin-ansible
AnsibleCLI: cache Galaxy collections and Python dependencies across executions
- Dominant language
- Java
- Stars
- 2
- Forks
- 6
- Avg merge
- 17h 5m
- Merged PRs (30d)
- 6
Description
## Summary
`AnsibleCLI` reinstalls Galaxy collections and Python packages on every task execution, with nothing carried over between runs. Give it first-class dependency caching, mirroring `dependencies` + `dependencyCacheEnabled` on the Python script tasks.
## Current behaviour
Since #74, the plugin auto-installs from files found in the working directory:
- `autoInstallGalaxyRequirements` (default `true`) runs `ansible-galaxy install -r requirements.yml`
- `autoInstallPythonRequirements` (default `true`) runs `pip install --no-cache-dir -r requirements.txt`
Both re-run in full on every execution. `--no-cache-dir` also disables pip's own HTTP cache, so nothing is reused even within the same worker. `ansible-galaxy` re-downloads collections into the working directory each time. `beforeCommands` has the same characteristic: it runs once per task execution and is not reused across the multiple ansible commands in a task, let alone across executions.
## Use case
Users declare a fixed set of Galaxy collections (plus the Python libs some of them need, e.g. `proxmoxer` for `community.proxmox`) so their playbooks can use them. The collection set changes rarely, but the install cost is paid on every single execution, which is the dominant part of runtime for short playbooks.
Two ways users work around this today:
- Pre-bake the collections into a custom `containerImage`. Works well, but each change needs an image build and, in regulated environments, an internal approval cycle. That approval cost is what makes this a non-starter for some users.
- Wrap `AnsibleCLI` in a `WorkingDirectory` task, have `beforeCommands` install into a subdirectory, and cache that subdirectory with a TTL ([caching docs](https://kestra.io/docs/concepts/caching)). This uses existing, supported features, but it requires restructuring the flow and hand-managing the cache path and TTL for something that is really a plugin-level concern.
## Proposal
Give Ansible the same treatment Python tasks already have:
- `dependencies` on `AnsibleCLI`: a list of collection/role specifiers (equivalent to the entries of a `requirements.yml`), so the set can be declared on the task instead of shipped as a file.
- `dependencyCacheEnabled` (default `true`): cache the resolved collections and their Python dependencies across executions, keyed on the resolved dependency set plus the container image, the way `io.kestra.plugin.scripts.python.Commands` already does.
Ideally the existing `autoInstall*` paths hit the same cache when `requirements.yml` / `requirements.txt` arrive via `namespaceFiles` or `inputFiles`, since that is how most users declare them today. Worth deciding whether the cache stores the installed collection tree or just the downloaded artifacts, given that some collections build native Python deps.
## Current Workaround
`WorkingDirectory` + `cache` with a TTL, or pre-baking into a custom container image. Both work; neither is first-class.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the AnsibleCLI entry point and compare its dependency installation paths with the Python script tasks' existing dependencies and dependencyCacheEnabled behavior. Review how autoInstallGalaxyRequirements and autoInstallPythonRequirements handle requirements.yml and requirements.txt. Done should include declared dependencies and reuse across executions, with the cache key and installed-versus-downloaded contents resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ansible, python
- Domain
- devops
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100