ansible-community / ansible-community/molecule-plugins

Podman driver fails with Error: reference parameter cannot be empty on Podman 5 when cmd_args is present

Open
#357 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
164
Forks
113
Avg merge
4d 20h
Merged PRs (30d)
1

Description

### Summary

When running Molecule with the Podman driver (`molecule-podman`) against Podman $5.0.3$, the *Create* step fails with:

```text
Error: reference parameter cannot be empty
exit code 125
```

This happens whenever the `cmd_args` parameter is present in the `containers.podman.podman_container` task in `playbooks/create.yml`, even if `cmd_args` should logically resolve to an empty list or a simple list of strings.

If I remove `cmd_args` entirely from that task, the scenario passes.

---

### Environment

- OS: (please fill in: macOS / Linux, version)
- Python: `3.13.12`
- `molecule`: `25.12.0`
- `molecule-podman`: `2.0.3`
- `ansible-core`: `2.20.3`
- Podman: `5.0.3`
- Collections: includes `containers.podman` (version as shipped with the above)

---

### Minimal `molecule.yml` (failing)

```yaml
---
dependency:
name: galaxy

driver:
name: podman

platforms:
- name: teleport-agent-test
image: "cgm-hub.oci.jfrog.cgm.ag/library/ubuntu:24.04"
pre_build_image: true
tty: true
extra_opts: [] # explicitly a list, to avoid odd values

provisioner:
name: ansible
env:
TPA_TELEPORT_TOKEN: ""
TPA_APT_USERNAME: ""
TPA_APT_PASSWORD: ""
TELEPORT_AGENT_LOGIN_IDENTITY_FILE: ""
inventory:
host_vars:
teleport-agent-test:
ansible_os_family: Debian
ansible_distribution: Ubuntu
ansible_distribution_release: jammy
ansible_hostname: test-host

verifier:
name: ansible
```

This is essentially a trivial Podman-based scenario just to bring up an Ubuntu container and run the role.

---

### Command

```bash
molecule --debug test -s default --no-parallel
```

---

### Error output (relevant part)

```text
TASK [Create molecule instance(s)] *********************************************
ok: [localhost] => (item=teleport-agent-test)

TASK [Wait for instance(s) creation to complete] *******************************
[ERROR]: Task failed: Module failed: Container teleport-agent-test exited with code 125 when runed
Origin: /.../site-packages/molecule_plugins/podman/playbooks/create.yml:179:7

failed: [localhost] (item=teleport-agent-test) => {
"msg": "Container teleport-agent-test exited with code 125 when runed",
"stderr": "Error: reference parameter cannot be empty\n",
"stdout": "",
...
}
```

Manual Podman usage works fine:

```bash
podman pull cgm-hub.oci.jfrog.cgm.ag/library/ubuntu:24.04

podman run --rm -d --name teleport-agent-test \
cgm-hub.oci.jfrog.cgm.ag/library/ubuntu:24.04 \
/bin/bash -c "sleep infinity"
# container starts successfully
```

So the image and Podman itself are fine.

---

### Relevant snippet from `playbooks/create.yml` (as shipped)

This is the `Create molecule instance(s)` task from `molecule-podman 2.0.3`:

```yaml
- name: Create molecule instance(s)
containers.podman.podman_container:
name: "{{ item.name }}"
cap_add: "{{ item.capabilities | default(omit) }}"
# it's necessary to exclude empty args as ansible modules run_command method doesn't handle this well
cmd_args: "{{ molecule_podman_args | reject('equalto', '') | list + item.extra_opts | default([]) }}"
command: "{{ (command_directives_dict | default({}))[item.name] | default('') }}"
detach: "{{ item.detach | default(omit) }}"
device: "{{ item.devices | default(omit) }}"
dns: "{{ item.dns_servers | default(omit) }}"
env: "{{ item.env | default(omit) }}"
etc_hosts: "{{ item.etc_hosts | default(omit) }}"
executable: "{{ podman_exec }}"
expose: "{{ item.exposed_ports | default(omit) }}"
hostname: "{{ item.hostname | default(omit) }}"
image: "{{ item.pre_build_image | default(false) | ternary('', 'molecule_local/') }}{{ item.image }}"
ip: "{{ item.ip | default(omit) }}"
network: "{{ item.network | default(omit) }}"
pid: "{{ item.pid_mode | default(omit) }}"
privileged: "{{ item.privileged | default(omit) }}"
publish: "{{ item.published_ports | default(omit) }}"
security_opt: "{{ item.security_opts | default(omit) }}"
systemd: "{{ item.systemd | default(omit) }}"
tmpfs: "{{ item.tmpfs | default(omit) }}"
tty: "{{ item.tty | default(omit) }}"
ulimits: "{{ item.ulimits | default(omit) }}"
volume: "{{ item.volumes | default(omit) }}"
state: started
register: server
changed_when: false
with_items: "{{ molecule_yml.platforms }}"
loop_control:
label: "{{ item.name }}"
async: 7200
poll: 0
vars:
molecule_podman_args:
- "{% if item.cgroup_manager is defined %}--cgroup-manager={{ item.cgroup_manager }}{% endif %}"
- >-
{% if item.restart_policy is defined %}
--restart={{ item.restart_policy }}{% if item.restart_retries is defined %}:{{ item.restart_retries }}{% endif %}
{% endif %}
- "{% if item.storage_opt is defined %}--storage-opt={{ item.storage_opt }}{% endif %}"
- "{% if item.storage_driver is defined %}--storage-driver={{ item.storage_driver }}{% endif %}"
```

---

### What I tested

1. **As-shipped `create.yml` with Podman 5.0.3 (failing)**
With the `cmd_args` line above present, `molecule test` fails on the “Wait for instance(s) creation to complete” task, reporting that the container exited with code 125 and `Error: reference parameter cannot be empty`.

2. **Simplified `podman_container` call (works)**
If I temporarily simplify the task to:

```yaml
- name: Create molecule instance(s)
containers.podman.podman_container:
name: "{{ item.name }}"
image: "{{ item.pre_build_image | default(false) | ternary('', 'molecule_local/') }}{{ item.image }}"
tty: "{{ item.tty | default(omit) }}"
state: started
register: server
changed_when: false
with_items: "{{ molecule_yml.platforms }}"
loop_control:
label: "{{ item.name }}"
async: 7200
poll: 0
vars:
molecule_podman_args: []
```

then `molecule test` **succeeds** consistently. So the core `containers.podman.podman_container` → Podman 5 path works.

3. **Reintroducing fields gradually**
I then reintroduced all the other parameters (`env`, `hostname`, volumes, network, pid, systemd, tmpfs, security, ulimits, etc.) one by one. All of them work together without error *until* I add back `cmd_args`.

4. **`cmd_args` is the trigger**
As soon as I add back:

```yaml
cmd_args: "{{ molecule_podman_args | reject('equalto', '') | list + item.extra_opts | default([]) }}"
```

the container exits with code 125 and Podman prints `Error: reference parameter cannot be empty` again.

5. **Trying a “safer” `cmd_args` expression — still fails**
I tried replacing the expression with a more explicit form to be sure about Jinja precedence and defaults:

```yaml
cmd_args: >-
{{
(
(molecule_podman_args | reject('equalto', '') | list)
+
(item.extra_opts | default([]))
)
}}
```

I also explicitly set `extra_opts: []` in `molecule.yml` (a proper empty list). Even with this change, the error still occurs.
If I comment out `cmd_args` entirely, the error disappears.

---

### Hypothesis

- With Podman 5.0.3, the combination of `containers.podman.podman_container` and a non-empty `cmd_args` (or possibly even `cmd_args=[]`) appears to generate a `podman run` invocation that Podman interprets as having an “empty reference” argument, hence the `Error: reference parameter cannot be empty`.
- The issue is not in the `image` parameter (it works fine when `cmd_args` is removed) and not in the other container options (env, volumes, network, systemd, etc.), but specifically tied to `cmd_args`.
- Given that simplifying the task to omit `cmd_args` makes the scenario pass, this seems to be either:
- a bug in how Molecule’s Podman driver constructs `cmd_args` and passes it to the module, or
- a compatibility issue between `containers.podman.podman_container` and Podman 5 when `cmd_args` is used.

---

### Request

- Is `cmd_args` known to work with Podman 5.x in this context?
- Could the `Create` task be adjusted to:
- skip setting `cmd_args` entirely when the computed list would be empty, and/or
- avoid patterns known to trigger “reference parameter cannot be empty” with Podman 5?
- Alternatively, is there a recommended version combination (Podman / `containers.podman` / `molecule-podman`) for Podman-based Molecule testing that avoids this issue?

I’m happy to provide:

- Full `molecule --debug` logs,
- `podman info`,
- or to test proposed changes to `create.yml` against my environment.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with molecule_plugins/podman/playbooks/create.yml, especially the Create molecule instance(s) task and its cmd_args expression. Run `molecule --debug test -s default --no-parallel` with Podman 5.0.3, compare the failing task with the simplified version that omits cmd_args, and inspect how the containers.podman.podman_container invocation handles the computed list. Done means the scenario starts successfully without Podman reporting an empty reference.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.