canonical / canonical/notebook-operators

jupyter-controller: crash-loops on startup when Istio VirtualService CRD is not installed

Open
#563 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6
Forks
12
Avg merge
3d 5h
Merged PRs (30d)
12

Description

**Note: This issue was generated with AI assistance (GitHub Copilot) based on automated log analysis and triage.**
Filed by @canonical/solutions-qa

---

## Summary

`jupyter-controller` (rev 1301, `1.10/stable`) enters a persistent crash loop when deployed in an environment without Istio. The controller fails to sync its caches because it attempts to set up a watch on `VirtualService.networking.istio.io` — which is not present — causing the process to exit, pebble to restart it, and the cycle to repeat every ~2 minutes until eventually (after ~10 attempts, ~20+ minutes) it succeeds. This exceeds the standard 15-minute deployment timeout in integration tests.

## Error

From `neighbor-0.log` (jupyter-controller pod) during test execution 460736:

```
ERROR controller-runtime.source if kind is a CRD, it should be installed before calling Start
{"kind": "VirtualService.networking.istio.io", "error": "no matches for kind \"VirtualService\" in version \"networking.istio.io/v1alpha3\""}

ERROR controller.notebook Could not wait for Cache to sync
{"reconciler group": "kubeflow.org", "reconciler kind": "Notebook",
"error": "failed to wait for notebook caches to sync: timed out waiting for cache to be synced"}

ERROR setup problem running manager
{"error": "failed to wait for notebook caches to sync: timed out waiting for cache to be synced"}

Service "jupyter-controller" on-failure action is "restart", waiting ~500ms before restart (backoff 1)
```

## Crash loop timeline

| Time (UTC) | Event |
|---|---|
| 08:32:41Z | jupyter-controller starts |
| 08:34:42Z | 1st crash → pebble restarts |
| 08:36:45Z | 2nd crash → pebble restarts |
| 08:38:48Z | 3rd crash → pebble restarts |
| 08:40:51Z | 4th crash → pebble restarts |
| 08:42:54Z | 5th crash → pebble restarts |
| 08:44:56Z | 6th crash → pebble restarts |
| 08:46:59Z | 7th crash → pebble restarts |
| 08:49:02Z | 8th crash → pebble restarts |
| 08:51:04Z | 9th crash → pebble restarts |
| 08:52:47Z | **finally active** (~20 min after first start) |

The test timeout (15 min) expired at ~08:44:57Z, before the charm recovered.

## Deployment context

- **Test bundle**: `opentelemetry-collector-k8s` (target) + `jupyter-controller` (neighbor) + supporting charms
- **No Istio** was present in the bundle — this is a minimal integration test environment
- `jupyter-controller` is used as a neighbor charm to exercise the `logging` relation with `opentelemetry-collector-k8s`
- The target charm (`opentelemetry-collector-k8s`) was **active and healthy** throughout; only the neighbor failed

## Environment

- `jupyter-controller` rev 1301, channel `1.10/stable`
- Juju 3.6.21, Kubernetes (`k8s-production`)
- No Istio / no `VirtualService` CRD installed

## Expected behaviour

`jupyter-controller` should start successfully and reach active state in a reasonable time (< 5 min) when Istio is not present. The `VirtualService` watch should be treated as optional/conditional — either skipped entirely when the CRD is absent, or handled such that the cache-sync timeout is not triggered.

## Test Observer UI

https://test-observer.canonical.com/#/charms/406485?testExecutionId=460736&testResultId=10457266

Contributor guide

Open the contributing guide

Research direction

Start with jupyter-controller startup and cache setup in the minimal integration bundle, reproducing the deployment without Istio or the VirtualService CRD. Confirm the controller reaches active within the expected timeout and the logging-relation integration test no longer fails because of repeated cache-sync crashes.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
backend, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.