bug: pods start erroring out after a couple of days of running

## Overview
A problem since a long time now, but if we run the pods for longer timeframes, then after a point the pods for chains go into `CrashLoopBackOff` stopping the chains and making the deployment completely useless.

## Potential fix
Having persistent volume claims for the pods, allow them to gracefully restart if and when required.

Start by reproducing the long-running chain pods and inspecting the Kubernetes deployment and storage configuration related to the CrashLoopBackOff. Done means pods can restart gracefully after running for multiple days, without stopping the chains or making the deployment unusable.

Written by the indexing model from the issue text.