Spline agent affecting databricks driver performance

Hi @wajda, @cerveada

We are using spline agent with databricks and sending lineage by http requests using the httpsdispatcher . We are using Azure function to collect the lineage. What we saw was, during high loads (and so high response times) on the function, if the agent is not able to establish connection to the gateway, it continues to retry every 2 mins. But during this time all operations on the cluster was hanged. I am attaching the logs here for your reference. We had to remove spline installation and restart the cluster to make it normal. Though we are working on improving the Azure function response time by correctly sizing it, but we want to know if we can do anything in the spline setting as well to stop retries if once the gate connection is failed. We plan to install spline on 100 clusters and do not want to lose business team's trust. Please Help!

[log4j-2023-09-12-08 (1).log](https://github.com/AbsaOSS/spline-spark-agent/files/12746044/log4j-2023-09-12-08.1.log)