prometheus-community / prometheus-community/PushProx

Error polling - EOF

Open
#193 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
811
Forks
154
PR merge metrics
No merged PRs in 30d

Description

up to yesterday evening (~2024-11-27 18:00:00 UT) my configuration

  • client (debian12)
    • PushProx
    • several exporter like Prometheus-node-exporter (from Debian repository)
  • server (debian12)
    • Prometheus (from Debian repository)
    • PushProx
      works perfectly

Without any changes at the configuration and the firewalls now I got in the log of the PushProxy on the client

pushprox-client[7039]: ts=2024-11-27T11:35:29.979Z caller=main.go:197 level=error msg="Error polling:" err="Post \"http://server:8082/poll\": EOF"

and of curse on the server some time out error due to this:

pushprox-proxy[33701]: ts=2024-11-27T11:40:29.429Z caller=main.go:179 level=error msg="Error scraping:" err="Timeout reached for \"http://client:9101/metrics\": context canceled" url=http://client:9101/metrics

I have check access to the ports and also with curl by

curl -d 'test' http://server:8082/poll

which is received on the server

Nov 27 12:48:04 server pushprox-proxy[33701]: ts=2024-11-27T11:48:04.779Z caller=coordinator.go:136 level=info msg=WaitForScrapeInstruction fqdn=test
Nov 27 12:48:04 server pushprox-proxy[33701]: ts=2024-11-27T11:48:04.779Z caller=main.go:148 level=info msg="Error WaitForScrapeInstruction:" err="request is expired"

I have also pulled the latest version form PushProxy now and checked the system times - same errors.

Sending to the PushProxyon the clientvia curl -d 'test' http://localhost:9101/polls generates a return of

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
....

My configs:

  • server
    • pushproxy: --web.listen-address=:8082
    • prometheus: /etc/prometheus/prometheus.yml
 - job_name: push_proxy
   proxy_url: http://localhost:8082/
   scrape_interval: 120s
   scrape_timeout: 120s
   tls_config:
     insecure_skip_verify: true
   static_configs:
     - targets: ['client:9101']  # push proxy (etc, metrics);  Presuming the FQDN of the client is "client".
     - targets: ['client:9732']  # nftables (/home)
     - targets: ['client:9633']  # smartctl (/home)
     - targets: ['client:9902']  # smartctl (/home)
  • client: --proxy-url=http:/server:8082/ --log.level=debug --metrics-addr=:9101

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Trace the polling failure from client main.go:197 through the server main.go:179 and coordinator.go:136, using the provided client and server configurations to reproduce the EOF and timeout logs. Review the polling and scrape interaction first; done means identifying the cause of the connection failure and documenting a verified resolution or reproduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
debian, go, prometheus
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.