Update upgrade guide for cos-lite to cos, when cos-lite is on microk8s

Hi Team,

recently I was evaluating if we could replace COS Lite in our (Data platform) setups with COS HA. As there is no dedicated guide for deploying COS on MicroK8s, I followed the recommendations in [Getting started with COS Lite and MicroK8s](https://documentation.ubuntu.com/observability/tutorial/installation/cos-lite-microk8s-sandbox/) and [Migrate from COS Lite to COS](https://documentation.ubuntu.com/observability/how-to/migrate-cos-lite-to-cos/) for testing.

This resulted in the deployment getting completely stuck. I reproduced this a couple of times, it always ends up with the same results.

Here are the relevant outputs:

```
$ $ juju status --relations --storage

Model Controller Cloud/Region Version SLA Timestamp
cos k8s microk8s/localhost 3.6.7 unsupported 17:21:32Z

App Version Status Scale Charm Channel Rev Address Exposed Message
alertmanager 0.27.0 active 1 alertmanager-k8s 1/stable 160 10.152.183.248 no
catalogue active 1 catalogue-k8s 1/stable 82 10.152.183.132 no
grafana 9.5.3 active 1 grafana-k8s 1/stable 143 10.152.183.27 no
grafana-agent 0.40.4 blocked 1 grafana-agent-k8s 1/stable 115 10.152.183.172 no Missing ['grafana-cloud-config']|['send-remote-write'] for metrics-endpoint; ['grafana-cloud-config']|['tracing'] for...
loki blocked 1/3 loki-coordinator-k8s 1/stable 25 10.152.183.65 no [consistency] Missing any worker relation.
loki-backend 3.0.0 blocked 1/3 loki-worker-k8s 1/stable 36 10.152.183.245 no Missing relation to a coordinator charm
loki-read 3.0.0 blocked 1/3 loki-worker-k8s 1/stable 36 10.152.183.104 no Missing relation to a coordinator charm
loki-s3-integrator active 1 s3-integrator 2/edge 157 10.152.183.133 no
loki-write 3.0.0 blocked 1/3 loki-worker-k8s 1/stable 36 10.152.183.47 no Missing relation to a coordinator charm
mimir blocked 1/3 mimir-coordinator-k8s 1/stable 39 10.152.183.122 no [consistency] Missing any worker relation.
mimir-backend 2.13.0 blocked 1/3 mimir-worker-k8s 1/stable 45 10.152.183.173 no Missing relation to a coordinator charm
mimir-read 2.13.0 blocked 1/3 mimir-worker-k8s 1/stable 45 10.152.183.230 no Missing relation to a coordinator charm
mimir-s3-integrator active 1 s3-integrator 2/edge 157 10.152.183.34 no
mimir-write 2.13.0 blocked 1/3 mimir-worker-k8s 1/stable 45 10.152.183.186 no Missing relation to a coordinator charm
self-signed-certificates active 1 self-signed-certificates 1/edge 319 10.152.183.151 no
tempo blocked 1/3 tempo-coordinator-k8s 1/stable 79 10.152.183.115 no [consistency] Missing any worker relation.
tempo-s3-integrator active 1 s3-integrator 2/edge 157 10.152.183.90 no
traefik 2.11.0 active 1 traefik-k8s latest/edge 236 10.152.183.222 no Serving at 10.241.0.21

Unit Workload Agent Address Ports Message
alertmanager/0* active idle 10.1.2.185
catalogue/0* active idle 10.1.2.178
grafana-agent/0* blocked idle 10.1.2.180 Missing ['grafana-cloud-config']|['send-remote-write'] for metrics-endpoint; ['grafana-cloud-config']|['tracing'] for...
grafana/0* active idle 10.1.2.166
loki-backend/0* blocked idle 10.1.2.136 Missing relation to a coordinator charm
loki-backend/1 waiting allocating installing agent
loki-backend/2 waiting allocating installing agent
loki-read/0* blocked idle 10.1.2.137 Missing relation to a coordinator charm
loki-read/1 waiting allocating installing agent
loki-read/2 waiting allocating installing agent
loki-s3-integrator/0* active idle 10.1.2.175
loki-write/0* blocked idle 10.1.2.165 Missing relation to a coordinator charm
loki-write/1 waiting allocating installing agent
loki-write/2 waiting allocating installing agent
loki/0* blocked idle 10.1.2.138 [consistency] Missing any worker relation.
loki/1 waiting allocating installing agent
loki/2 waiting allocating installing agent
mimir-backend/0* blocked idle 10.1.2.155 Missing relation to a coordinator charm
mimir-backend/1 waiting allocating installing agent
mimir-backend/2 waiting allocating installing agent
mimir-read/0* blocked idle 10.1.2.162 Missing relation to a coordinator charm
mimir-read/1 waiting allocating installing agent
mimir-read/2 waiting allocating installing agent
mimir-s3-integrator/0* active idle 10.1.2.176
mimir-write/0* blocked idle 10.1.2.171 Missing relation to a coordinator charm
mimir-write/1 waiting allocating installing agent
mimir-write/2 waiting allocating installing agent
mimir/0* blocked idle 10.1.2.187 [consistency] Missing any worker relation.
mimir/1 waiting allocating installing agent
mimir/2 waiting allocating installing agent
self-signed-certificates/0* active idle 10.1.2.173
tempo-s3-integrator/0* active idle 10.1.2.183
tempo/0* blocked idle 10.1.2.189 [consistency] Missing any worker relation.
tempo/1 waiting allocating installing agent
tempo/2 waiting allocating installing agent
traefik/0* active idle 10.1.2.179 Serving at 10.241.0.21

Offer Application Charm Rev Connected Endpoint Interface Role
alertmanager-karma-dashboard alertmanager alertmanager-k8s 160 0/0 karma-dashboard karma_dashboard provider
grafana-dashboards grafana grafana-k8s 143 0/0 grafana-dashboard grafana_dashboard requirer

Integration provider Requirer Interface Type Message
alertmanager:grafana-dashboard grafana:grafana-dashboard grafana_dashboard regular
alertmanager:grafana-source grafana:grafana-source grafana_datasource regular
alertmanager:replicas alertmanager:replicas alertmanager_replica peer
alertmanager:self-metrics-endpoint grafana-agent:metrics-endpoint prometheus_scrape regular
catalogue:catalogue alertmanager:catalogue catalogue regular
catalogue:catalogue grafana:catalogue catalogue regular
catalogue:replicas catalogue:replicas catalogue_replica peer
grafana-agent:peers grafana-agent:peers grafana_agent_replica peer
grafana-agent:tracing-provider grafana:charm-tracing tracing regular
grafana:grafana grafana:grafana grafana_peers peer
grafana:replicas grafana:replicas grafana_replicas peer
loki-s3-integrator:s3-credentials loki:s3 s3 regular
mimir-s3-integrator:s3-credentials mimir:s3 s3 regular
tempo:peers tempo:peers tempo_peers peer
traefik:ingress alertmanager:ingress ingress regular
traefik:ingress catalogue:ingress ingress regular
traefik:peers traefik:peers traefik_peers peer
traefik:traefik-route grafana:ingress traefik_route regular

Storage Unit Storage ID Type Pool Mountpoint Size Status Message
alertmanager/0 data/0 filesystem kubernetes /var/lib/juju/storage/data/0 1.0 GiB attached Successfully provisioned volume pvc-5c441826-cc45-43fe-9187-4b8b13590897
grafana-agent/0 data/5 filesystem kubernetes /var/lib/juju/storage/data/0 1.0 GiB attached Successfully provisioned volume pvc-9eb7b95d-4724-4fab-88ad-0b0a6ab28ec7
grafana/0 database/8 filesystem kubernetes /var/lib/juju/storage/database/0 1.0 GiB attached Successfully provisioned volume pvc-2194de6d-22f7-47fc-a2f6-45a0b3dece58
loki-backend/0 loki-persisted/11 filesystem kubernetes /var/lib/juju/storage/loki-persisted/0 1.0 GiB attached Successfully provisioned volume pvc-b28a0312-da6a-4ea0-b245-3e973d30e2d3
loki-backend/1 loki-persisted/13 filesystem pending
loki-backend/2 loki-persisted/15 filesystem pending
loki-read/0 loki-persisted/12 filesystem kubernetes /var/lib/juju/storage/loki-persisted/0 1.0 GiB attached Successfully provisioned volume pvc-5f649522-75af-4028-91b2-1ec057bc1b71
loki-read/1 loki-persisted/14 filesystem pending
loki-read/2 loki-persisted/16 filesystem pending
loki-write/0 loki-persisted/17 filesystem kubernetes /var/lib/juju/storage/loki-persisted/0 1.0 GiB attached Successfully provisioned volume pvc-56e41132-e4ee-4a30-9d72-12ad984b7568
loki-write/1 loki-persisted/18 filesystem pending
loki-write/2 loki-persisted/19 filesystem pending
mimir-backend/0 data/32 filesystem kubernetes /var/lib/juju/storage/data/0 1.0 GiB attached Successfully provisioned volume pvc-433ec3ed-a4b1-4932-bb9d-45067729ade3
mimir-backend/0 recovery-data/33 filesystem kubernetes /var/lib/juju/storage/recovery-data/0 1.0 GiB attached Successfully provisioned volume pvc-6969e257-60a6-4714-a6e3-df6dfea07e51
mimir-backend/1 data/34 filesystem pending
mimir-backend/1 recovery-data/35 filesystem pending
mimir-backend/2 data/36 filesystem pending
mimir-backend/2 recovery-data/37 filesystem pending
mimir-read/0 data/20 filesystem kubernetes /var/lib/juju/storage/data/0 1.0 GiB attached Successfully provisioned volume pvc-83f578a5-2484-4a0d-9515-064e1a2846ce
mimir-read/0 recovery-data/21 filesystem kubernetes /var/lib/juju/storage/recovery-data/0 1.0 GiB attached Successfully provisioned volume pvc-15ccd287-d68f-439c-94e2-602983f7ccca
mimir-read/1 data/22 filesystem pending
mimir-read/1 recovery-data/23 filesystem pending
mimir-read/2 data/24 filesystem pending
mimir-read/2 recovery-data/25 filesystem pending
mimir-write/0 data/26 filesystem kubernetes /var/lib/juju/storage/data/0 1.0 GiB attached Successfully provisioned volume pvc-92fc2662-0b65-4aaf-9159-2e85934d00af
mimir-write/0 recovery-data/27 filesystem kubernetes /var/lib/juju/storage/recovery-data/0 1.0 GiB attached Successfully provisioned volume pvc-741dd3e7-fab1-42d6-b1ba-90d941e4c200
mimir-write/1 data/28 filesystem pending
mimir-write/1 recovery-data/29 filesystem pending
mimir-write/2 data/30 filesystem pending
mimir-write/2 recovery-data/31 filesystem pending
tempo/0 data/7 filesystem kubernetes /tempo-data 1.0 GiB attached Successfully provisioned volume pvc-97f71dd6-8db2-4d53-ad26-6f93f0ffa78e
tempo/1 data/9 filesystem pending
tempo/2 data/10 filesystem pending
traefik/0 configurations/6 filesystem kubernetes /var/lib/juju/storage/configurations/0 1.0 GiB attached Successfully provisioned volume pvc-a1d84648-e3f1-4773-8566-abd49253e1c7
```

```
$ microk8s.kubectl events -n cos -w
LAST SEEN TYPE REASON OBJECT MESSAGE
4m46s (x161 over 44m) Normal WaitForPodScheduled PersistentVolumeClaim/loki-backend-loki-persisted-a1f46d61-loki-backend-1 waiting for pod loki-backend-1 to be scheduled
4m46s (x161 over 44m) Normal WaitForPodScheduled PersistentVolumeClaim/loki-backend-loki-persisted-a1f46d61-loki-backend-2 waiting for pod loki-backend-2 to be scheduled
4m46s (x161 over 44m) Normal WaitForPodScheduled PersistentVolumeClaim/loki-read-loki-persisted-40b6e960-loki-read-1 waiting for pod loki-read-1 to be scheduled
4m46s (x161 over 44m) Normal WaitForPodScheduled PersistentVolumeClaim/loki-read-loki-persisted-40b6e960-loki-read-2 waiting for pod loki-read-2 to be scheduled
4m46s (x161 over 44m) Normal WaitForPodScheduled PersistentVolumeClaim/loki-write-loki-persisted-0b6e10f5-loki-write-1 waiting for pod loki-write-1 to be scheduled
4m46s (x161 over 44m) Normal WaitForPodScheduled PersistentVolumeClaim/loki-write-loki-persisted-0b6e10f5-loki-write-2 waiting for pod loki-write-2 to be scheduled
3m1s (x161 over 43m) Normal WaitForPodScheduled PersistentVolumeClaim/mimir-backend-data-b38ca6b7-mimir-backend-1 waiting for pod mimir-backend-1 to be scheduled
3m1s (x161 over 43m) Normal WaitForPodScheduled PersistentVolumeClaim/mimir-backend-data-b38ca6b7-mimir-backend-2 waiting for pod mimir-backend-2 to be scheduled
3m1s (x161 over 43m) Normal WaitForPodScheduled PersistentVolumeClaim/mimir-backend-recovery-data-b38ca6b7-mimir-backend-1 waiting for pod mimir-backend-1 to be scheduled
3m1s (x161 over 43m) Normal WaitForPodScheduled PersistentVolumeClaim/mimir-backend-recovery-data-b38ca6b7-mimir-backend-2 waiting for pod mimir-backend-2 to be scheduled
4m46s (x161 over 44m) Normal WaitForPodScheduled PersistentVolumeClaim/mimir-read-data-7ab6ba2a-mimir-read-1 waiting for pod mimir-read-1 to be scheduled
4m46s (x161 over 44m) Normal WaitForPodScheduled PersistentVolumeClaim/mimir-read-data-7ab6ba2a-mimir-read-2 waiting for pod mimir-read-2 to be scheduled
4m46s (x161 over 44m) Normal WaitForPodScheduled PersistentVolumeClaim/mimir-read-recovery-data-7ab6ba2a-mimir-read-1 waiting for pod mimir-read-1 to be scheduled
4m46s (x161 over 44m) Normal WaitForPodScheduled PersistentVolumeClaim/mimir-read-recovery-data-7ab6ba2a-mimir-read-2 waiting for pod mimir-read-2 to be scheduled
4m46s (x161 over 44m) Normal WaitForPodScheduled PersistentVolumeClaim/mimir-write-data-108e8a4a-mimir-write-1 waiting for pod mimir-write-1 to be scheduled
4m46s (x161 over 44m) Normal WaitForPodScheduled PersistentVolumeClaim/mimir-write-data-108e8a4a-mimir-write-2 waiting for pod mimir-write-2 to be scheduled
4m46s (x161 over 44m) Normal WaitForPodScheduled PersistentVolumeClaim/mimir-write-recovery-data-108e8a4a-mimir-write-1 waiting for pod mimir-write-1 to be scheduled
4m46s (x161 over 44m) Normal WaitForPodScheduled PersistentVolumeClaim/mimir-write-recovery-data-108e8a4a-mimir-write-2 waiting for pod mimir-write-2 to be scheduled
4m46s (x161 over 44m) Normal WaitForPodScheduled PersistentVolumeClaim/tempo-data-2ac34073-tempo-1 waiting for pod tempo-1 to be scheduled
4m46s (x161 over 44m) Normal WaitForPodScheduled PersistentVolumeClaim/tempo-data-2ac34073-tempo-2 waiting for pod tempo-2 to be scheduled
106s (x10 over 44m) Normal nodeAssigned Service/traefik-lb announcing from node "bunsen" with protocol "layer2"
```

It can not be a resource issue, as my deployment was on a machine with these specs
```
32 CPU cores
126 GB RAM
820 GB disk space
```

My deployment was on:
- MicroK8s
- 1.32-strict
- Add-ons: dns, hostpath-storage, metallb
- Juju
- 3.6.5
- Microceph
- 19.2.0
- Rados-GW enabled (no TLS)
- S3 storage buckets created before COS deployment: “tempo”, “mimir”, “loki”

I deployed using the following terraform module (following your guidelines from the COS module):
```
$ cat main.tf
terraform {
required_version = ">= 1.5"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.0"
}
juju = {
source = "juju/juju"
version = "0.19.0"
}
null = {
source = "hashicorp/null"
version = "~> 3.0"
}
}
}

# COS module that deploy the whole Canonical Observability Stack
module "cos" {
source = "git::https://github.com/canonical/observability-stack//terraform/cos"
model = "cos"
channel = "1/stable"
s3_integrator_channel = "2/edge"
ssc_channel = "1/edge"
traefik_channel = "latest/edge"
cloud = "self-managed"
use_tls = true
s3_endpoint = "http://10.241.0.21:80"
s3_secret_key = XXX
s3_access_key = XXX
loki_bucket = "loki"
mimir_bucket = "mimir"
tempo_bucket = "tempo"
loki_coordinator_units = 3
loki_backend_units = 3
loki_read_units = 3
loki_write_units = 3
mimir_coordinator_units = 3
mimir_backend_units = 3
mimir_read_units = 3
mimir_write_units = 3
tempo_coordinator_units = 3
tempo_compactor_units = 3
tempo_distributor_units = 3
tempo_ingester_units = 3
tempo_metrics_generator_units = 3
tempo_querier_units = 3
tempo_query_frontend_units = 3
alertmanager_revision = null
catalogue_revision = null
grafana_revision = null
grafana_agent_revision = null
loki_coordinator_revision = null
loki_worker_revision = null
mimir_coordinator_revision = null
mimir_worker_revision = null
ssc_revision = null
s3_integrator_revision = 157 # FIXME: https://github.com/canonical/observability/issues/342
tempo_coordinator_revision = null
tempo_worker_revision = null
traefik_revision = null
}
```

Could you please investigate why deploying COS HA fails on MicroK8s, or provide guidance on how to deploy?