"WaitForScrapeInstruction" errors

Hi all,

I am seeing several msg="Error WaitForScrapeInstruction messages in the server log, with corresponding msg="Error reading request:" err="invalid method "<!DOCTYPE"" messages on the client log.

As an example, see the below traces:

Server:
Aug 08 16:11:32 hosting.metersense.com proxy[13178]: level=info ts=2019-08-08T20:11:32.822Z caller=coordinator.go:123 msg=WaitForScrapeInstruction fqdn=myclient.harrissmartworks.com

Aug 08 16:11:32 hosting.metersense.com proxy[13178]: level=info ts=2019-08-08T20:11:32.822Z caller=proxy.go:129 msg="Error WaitForScrapeInstruction:" err="request is expired"

Client:
level=error ts=2019-08-08T20:11:31.805Z caller=client.go:167 msg="Error reading request:" err="invalid method \"<!DOCTYPE\""

(The client and server are synchronized to within a couple seconds of each other, with a 4-hour timezone difference.)

These errors are present every minute -- except on those minutes when the actual scrape request comes in, which is handled successfully. The current scrape interval is 10m.

I also noticed there is no "WaitForScrapeInstruction" method defined in client.go, but am unable to tell if that is the cause for the errors.

I logged #56 recently, and while I cannot discard the possibility that the two issues are connected, I thought it best to keep them separate for now. As mentioned there I am running the latest version of both the client and proxy software.

Start by tracing the log sites named in coordinator.go:123, proxy.go:129, and client.go:167, then compare the recurring request timing with the configured 10m scrape interval. Reproduce or analyze the invalid-method and expired-request sequence, and document or fix the behavior so normal scrape intervals no longer produce these errors.