canonical / canonical/charm-integration-testing

bundle_builder_x: unbounded domain expansion causes solver timeouts/failures even for minimal, fully-specified specs

Open
#792 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6
Forks
1
Avg merge
1d 22h
Merged PRs (30d)
96

Description

## Summary

While investigating #726 (`vault-k8s:vault-pki/tls-certificates/pgbouncer-k8s:certificates` bundle bloat), I found and fixed a specific interface-collision bug in `pgbouncer-k8s.yaml` that caused daisy-chaining. However, the underlying "bundle never converges / grows every run" symptom also reproduces independently of that bug, pointing to a deeper, pre-existing issue in `bundle_builder_x`'s core domain-expansion algorithm.

## Reproduction

Even a minimal, fully-specified two-charm spec with no ambiguity in endpoint resolution triggers massive, unbounded domain expansion:

```yaml
---
models:
- name: target-model
platform: kubernetes
applications:
target:
charm: pgbouncer-k8s
channel: "1/stable"
db:
charm: postgresql-k8s
channel: "14/stable"
integrations:
- application: target
endpoint: backend-database
remote_application: db
remote_endpoint: database
```

Run:
```
poetry run bundle-builder-x --spec spec.yaml --overrides ./static/charm-overrides/ --log-level INFO
```

Expected: the solver resolves the two explicitly-specified charms and their mandatory secondary endpoints (certificates, observability, etc.) and produces a bundle.

Observed: the solver's "expand domain" loop pulls in a large number of apparently unrelated charms across iterations - e.g. `airbyte-k8s`, `discourse-k8s`, `glauth-k8s`, `grafana-k8s`, `hockeypuck-k8s`, `httprequest-lego-provider`, `hydra`, `juju-jimm-k8s`, `kyuubi-k8s`, `netbox-k8s`, `ranger-k8s`, `superset-k8s`, `temporal-k8s`, `temporal-worker-k8s`, `spark-integration-hub-k8s` - none of which have any relation to `pgbouncer-k8s` or `postgresql-k8s` in the spec. After 5-6 iterations the domain size and constraint count grow enough that the solver either hits `Solver timed out after 0:01:00` or `Cannot expand domain to handle failed assertion tags` and the build fails outright.

This reproduces identically on a plain single-charm spec (`pgbouncer-k8s` alone, no explicit dependencies at all), so it is not specific to `pgbouncer-k8s` or to the interface-collision bug fixed in #726 - it appears to be a general property of the domain-expansion search once the charm-overrides directory reaches its current size/interconnectedness.

## Impact

This matches the scope-growth complaint in #726 ("bundle grew from 5 to 27 applications... generated bundle... 0% pass rate") and is very likely affecting other test plans beyond the one reported there, since it is triggered by ordinary, common interfaces (`postgresql_client`) rather than anything charm-specific.

## Suspected area

`bundle_builder_x/bundle_builder_x/bundle_builder.py` - the iterative "expand domain" / unsat-core-driven charm-discovery loop (`_iterative_descent`, `_try_z3_optimize`, and the code that adds new candidate charms in response to failed assertion tags). Needs investigation into:
- Whether charm candidates are being added too eagerly/broadly in response to unrelated unsat-core entries.
- Whether there should be a bound on domain size, or a way to scope candidate search to charms actually reachable from endpoints in the original spec.
- Whether per-iteration solver timeouts are masking a correctness issue vs. a pure performance/scaling issue.

## Environment

- Repo state: `main` @ commit with the `pgbouncer-k8s.yaml` "pooled" feature fix from #726 applied (also reproduces without it).
- `bundle-builder-x` CLI, default `--charmhub-url` (production Charmhub API).

## Related

- #726 - the interface-collision fix there does not resolve this issue; both were investigated together but are separate root causes.

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with the two-charm spec using the documented poetry run bundle-builder-x command. Inspect bundle_builder_x/bundle_builder_x/bundle_builder.py, especially _iterative_descent, _try_z3_optimize, and the failed-assertion candidate-expansion path. Done means the minimal and single-charm cases no longer expand indefinitely and complete without solver timeout or domain-expansion failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, tooling
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.