"malformed HTTP request \"\""

Hi!
I'm having troubles running PushProx on my staging server.

The most informative error I see is from proxy client receiving bad request:

$ ./pushprox-client --proxy-url=http://prom-proxy-stg.***.com/ --fqdn=`hostname` --metrics-addr=":9368"
level=info ts=2021-09-22T10:26:52.905Z caller=main.go:245 msg="URL and FQDN info" proxy_url=http://prom-proxy-stg.***.com/ fqdn=argon-u1910
level=error ts=2021-09-22T10:27:53.467Z caller=main.go:205 msg="Error reading request:" err="malformed HTTP request \"\""
level=error ts=2021-09-22T10:28:55.319Z caller=main.go:205 msg="Error reading request:" err="malformed HTTP request \"\""

And then probably because of that pushprox-proxy fails to retrieve data from the client:

caller=coordinator.go:137 msg=WaitForScrapeInstruction fqdn=argon-u1910
caller=main.go:149 msg="Error WaitForScrapeInstruction:" err="request is expired"

Some more details:

The client gets successfully registered at proxy/clients page: [{"targets":["argon-u1910"],"labels":null}]

And Prometheus is able to see that client live, but fails to scrape.

Prometheus config files and dashboard screenshots:

If I run proxy-server locally - everything is fine. So the problem is probably somewhere in the stack on which I'm trying to host it on cloud my servers. Should there be some additional ports open or something? Might the problem be in some http/https redirect where the request body is lost?

Any help will be much appreciated!

Thank you!

Start with the client error at main.go:205 and the proxy wait path at coordinator.go:137, then compare the cloud deployment with the locally working proxy-server setup. Trace the request between pushprox-client and pushprox-proxy, including the HTTP/HTTPS redirect or required port exposure mentioned in the report. Done means identifying why the client request is malformed and restoring successful Prometheus scrapes.