VCL not updated on specific circumstances

Describe the bug
Already a longer time, I had a suspicion, that sometimes the VCL in the cluster is not updated.
Now I could reproduce (or better say: spot and document occurence).
It seems to be as following:

• watch loop picks the change (after config map with VCL in the cluster has been updated and via mapped volume is updated in the running pod)
• change is set to internal data structure of the controller
• watch.go rebuildConfig() tries to set the rendered VCL to the varnish via CLI
• something fails and rebuildConfig() returns error - error is logged (as warning)
• as the new VCL is already updated in the data structure of the controller, it will never be retried and except for the present log message silently ignored - cluster pod will further run with the old config

Attachment screenshots.

To Reproduce
Unfortunately, I can't reproduce, as it happens semi randomly. Probably it could be reproduced by applying some stress to the varnish so the "varnish child is terminated/dies during VCL update".

Expected behavior
VCL is updated (or at least retried few times on fail)

Environment:

• Kubernetes version: 1.24
• kube-httpcache version: master branch

Configuration
-varnish-vcl-template=/etc/varnish/tmpl/frontend-tmpl.vcl # mapped from config map
-varnish-vcl-template-poll=true

Additional context
I did further analysis. In my case, it seems the following happens:

• varnish daemon gets request via CLI
• varnish master process can't communicate with its child process and terminates the child process and restarts it - note that this is one specific case - it could be any transient error why the update fails
• varnish master via CLI returns fail (to the kube-httpcache)
• VCL is not updated

Preliminary fix idea:

• in watch.go replace: errors <- v.rebuildConfig(ctx)
• with something like: v.rebuildConfigWithRetry(ctx, errors)
• new function would call v.rebuildConfig(ctx) and retry the rebuild after error

If I find some time, I can try to implement the fix, but unsure when.
Example of error situation:


The line which begins with "W" is from kube-httpcache, rest without prefix come from the varnish process itself.
Note that we run the pods with increased log level -v=7

Note that in our case probably the problem could be mitigated by fine-tuning varnish itself, so that the child "never dies". And it could be questionable if it is a bug or feature (as the error is already logged as warning). But I think some kind of retry logic would be useful to make the VCL update more resilient in the cases where varnish VCL update fails, but the failure is transient and would succeed on retry. Because even if it happens on quite rare occasions, the result is part of the cluster running with old config, which could be very dangerous.

Start in watch.go at the update path where errors are received from rebuildConfig(ctx), then inspect rebuildConfig and how the controller stores the new VCL. Determine how a failed VCL update can be retried without silently losing the pending change. Done means transient rebuild failures are retried and the VCL eventually updates, with the failure still logged.