Config Server cannot pull from remote repo

Hi,

After almost 3 months without any issues the cloud config server suddenly failed pulling from the repo.
This happened silently on Saturday and on Monday we found out after checking the logs that the new config changes were not being picked from the config-server.

Exception thrown: Cannot pull from remote https://repo.git, the working tree is not clean.

No changes were pushed to the repo or new version of cloud server deployed anytime close to the silent crash. The good thing is that the issue can be resolved just by restarting the cloud config server(s). After that everything goes back to normal but now the question is how to monitor and get notified for such issues. It would be nice if there is a HealthIndicator reporting the health of the config server taking into consideration such state or/and the logging level changed to ERROR.

Detailed exception:

2016-12-10 03:47:09,118 [http-nio-8888-exec-5] INFO  o.s.c.a.AnnotationConfigApplicationContext Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@498613ae: startup date [Sat Dec 10 03:47:09 CET 2016]; root of context hierarchy
2016-12-10 03:47:09,123 [http-nio-8888-exec-5] INFO  o.s.c.c.s.e.NativeEnvironmentRepository Adding property source: file:/tmp/config-repo-7239745589329148613/application.properties
2016-12-10 03:47:09,123 [http-nio-8888-exec-5] INFO  o.s.c.a.AnnotationConfigApplicationContext Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@498613ae: startup date [Sat Dec 10 03:47:09 CET 2016]; root of context hierarchy
2016-12-10 03:48:08,949 [http-nio-8888-exec-10] INFO  o.s.c.c.s.e.MultipleJGitEnvironmentRepository Cannot pull from remote https://repo.git, the working tree is not clean.
2016-12-10 03:48:08,983 [http-nio-8888-exec-10] INFO  o.s.c.a.AnnotationConfigApplicationContext Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@11af7f6b: startup date [Sat Dec 10 03:48:08 CET 2016]; root of context hierarchy
2016-12-10 03:48:08,988 [http-nio-8888-exec-10] INFO  o.s.c.c.s.e.NativeEnvironmentRepository Adding property source: file:/tmp/config-repo-7239745589329148613/application.properties
2016-12-10 03:48:08,988 [http-nio-8888-exec-10] INFO  o.s.c.a.AnnotationConfigApplicationContext Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@11af7f6b: startup date [Sat Dec 10 03:48:08 CET 2016]; root of context hierarchy
2016-12-10 03:49:08,954 [http-nio-8888-exec-8] INFO  o.s.c.c.s.e.MultipleJGitEnvironmentRepository Cannot pull from remote https://repo.git, the working tree is not clean.
2016-12-10 03:49:08,981 [http-nio-8888-exec-8] INFO  o.s.c.a.AnnotationConfigApplicationContext Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@7f1733bd: startup date [Sat Dec 10 03:49:08 CET 2016]; root of context hierarchy

Thanks

Start by tracing the failed pull in MultipleJGitEnvironmentRepository and the config-server health reporting around the reported working-tree exception. Review the logs showing NativeEnvironmentRepository fallback behavior and the issue discussion. Done means the repository state is surfaced through health or error reporting so silent failures can be detected and acted on.