canonical / canonical/knative-operators
knative-operator goes to Error during remove event
- Dominant language
- Jinja
- Stars
- 2
- Forks
- 3
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 2
Description
During application removal, knative operator enters error state with juju status:
```
knative-operator/0* error idle 10.1.208.139 hook failed: "remove"
```
and debug logs of:
```
2023-03-07 20:51:34 INFO juju-log The OpenTelemetry Collector may not be deployed yet.This may be temporary or due to a missing otel-collector relation.
2023-03-07 20:51:34 INFO juju.worker.uniter.operation runhook.go:146 ran "update-status" hook (via hook dispatching script: dispatch)
2023-03-07 20:52:01 INFO juju-log The OpenTelemetry Collector may not be deployed yet.This may be temporary or due to a missing otel-collector relation.
2023-03-07 20:52:02 INFO juju-log Rendering manifests
2023-03-07 20:52:05 INFO juju-log Reconcile completed successfully
2023-03-07 20:52:05 INFO juju.worker.uniter.operation runhook.go:146 ran "config-changed" hook (via hook dispatching script: dispatch)
2023-03-07 20:52:06 INFO juju-log The OpenTelemetry Collector may not be deployed yet.This may be temporary or due to a missing otel-collector relation.
2023-03-07 20:52:06 INFO juju-log Rendering manifests
2023-03-07 20:52:08 INFO juju-log Reconcile completed successfully
2023-03-07 20:52:09 INFO juju.worker.uniter.operation runhook.go:146 ran "config-changed" hook (via hook dispatching script: dispatch)
2023-03-07 20:53:13 INFO juju.worker.caasunitterminationworker worker.go:82 terminating due to SIGTERM
2023-03-07 20:53:13 WARNING juju.worker.uniter.operation leader.go:123 we should run a leader-deposed hook here, but we can't yet
2023-03-07 20:53:14 INFO juju-log The OpenTelemetry Collector may not be deployed yet.This may be temporary or due to a missing otel-collector relation.
2023-03-07 20:53:14 INFO juju.worker.uniter.operation runhook.go:146 ran "stop" hook (via hook dispatching script: dispatch)
^[[A^[[B2023-03-07 20:53:15 INFO juju-log The OpenTelemetry Collector may not be deployed yet.This may be temporary or due to a missing otel-collector relation.
2023-03-07 20:53:15 INFO juju-log Rendering manifests
2023-03-07 20:53:16 INFO juju-log Rendering manifests
2023-03-07 20:53:17 WARNING juju-log Log string greater than 131071. Splitting into multiple chunks:
2023-03-07 20:53:17 WARNING juju-log Failed to delete resources: ... TRUNCATED ... with: deployments.apps "otel-collector" not found
2023-03-07 20:53:17 ERROR juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/lightkube/core/generic_client.py", line 188, in raise_for_status
resp.raise_for_status()
File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/httpx/_models.py", line 745, in raise_for_status
raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '404 Not Found' for url 'https://10.152.183.1/apis/apps/v1/namespaces/kubeflow/deployments/otel-collector'
For more information check: https://httpstatuses.com/404
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./src/charm.py", line 287, in
main(KnativeOperatorCharm)
File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/ops/main.py", line 438, in main
_emit_charm_event(charm, dispatcher.event_name)
File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/ops/main.py", line 150, in _emit_charm_event
event_to_emit.emit(*args, **kwargs)
File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/ops/framework.py", line 355, in emit
framework._emit(event) # noqa
File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/ops/framework.py", line 856, in _emit
self._reemit(event_path)
File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/ops/framework.py", line 931, in _reemit
custom_handler(event)
File "./src/charm.py", line 282, in _on_remove
raise e
File "./src/charm.py", line 277, in _on_remove
delete_many(
File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/charmed_kubeflow_chisme/lightkube/batch/_many.py", line 97, in delete_many
client.delete(res=obj.__class__, name=obj.metadata.name, namespace=namespace)
File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/lightkube/core/client.py", line 73, in delete
return self._client.request("delete", res=res, name=name, namespace=namespace)
File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/lightkube/core/generic_client.py", line 245, in request
return self.handle_response(method, resp, br)
File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/lightkube/core/generic_client.py", line 196, in handle_response
self.raise_for_status(resp)
File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/lightkube/core/generic_client.py", line 190, in raise_for_status
raise transform_exception(e)
lightkube.core.exceptions.ApiError: deployments.apps "otel-collector" not found
2023-03-07 20:53:17 ERROR juju.worker.uniter.operation runhook.go:140 hook "remove" (via hook dispatching script: dispatch) failed: exit status 1
```
Contributor guide
Research direction
Start in src/charm.py around the _on_remove handler and the delete_many call shown in the traceback. Reproduce removal when the otel-collector deployment is already absent, then verify the remove hook completes successfully without treating the Kubernetes 404 as an uncaught failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100