[STAC-25586] Test and document how to run Observability on Longhorn

STAC-25586

I’ve seen another example of running observability on Longhorn (a 4000-ha profile) where both are configured for HA:

• Longhorn 3 replicas of data
• Hadoop 3 replicas (total of 9 copies of all data)
• Kafka, Elastic, ClickHouse, Zookeeper all have 2 replicas (total of 6 copies of the data)

That is all great and safe, but having that many replicas is also slow and that causes problems, it also makes large storage requirements even bigger.

Can we simply recommend running on Longhorn with 1 replica in Longhorn only when using HA in Observability?

Longhorn is the main example here and we can document it in detail.

Other storage solutions may also be replicating the data and we should include the same recommendation without testing and documenting them all (recommendation in the form of: avoid additional data replication at the storage level).

This seems to be the interesting part from the Longhorn documentation: https://longhorn.io/docs/1.12.1/best-practices/#io-performance .

Specifically reducing to 2 replicas for performance, but even more so using strict-local mode for distributed databases (which requires only a single replica). Via node-affinity on the PV the statefulset pods will get scheduled always back to the same node (similar to how this works in AWS with EBS volumes, however there it is not tied to a node but to an AZ). When a node fails and a volume is lost this will require manual removal of the PVC for which data was lost to recover the pod (the pod will not start up anymore, because it is fixed to that node).

Suggested approach to test is to use a storage class with 1 replica and strict-local data locality (something like this, but make sure it matches the standard Longhorn storage class for other parameters).

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-database-sc
provisioner: driver.longhorn.io
allowVolumeExpansion: true
parameters:
  numberOfReplicas: "1"
  dataLocality: "strict-local"

Then verify that we can recover from a node failure after doing the manual intervention to remove the PVCs that were stored on the failed node.

No repository files or tests are named. Start with the proposed Longhorn StorageClass using one replica and strict-local data locality, then verify recovery after a node failure and the stated manual PVC removal. Done means documenting the tested Longhorn setup and the recommendation to avoid additional storage-level replication for other solutions.