Error on ingesting samples that are too old or are too far into the future

Hi!

Faced strange situation with timestamps.

Briefly:

1. If timestamp is taken from datetime.datetime.now() - everything is ok.
2. If timestamp is parsed from the date which looks same - I got warning in Prometheus server log about bad datetime and metric is not shown.

Here is code example with comments about which metrics works and which don't:

import datetime
import sys
import time
import pytz

from prometheus_client import (
    start_http_server,
)
from prometheus_client.core import (
    GaugeMetricFamily,
    REGISTRY,
)


class BugDemoMetricsCollector:

    def collect(self):
        dt_format = '%Y-%m-%d_%H-%M-%S.%f %z'
        dt_now = datetime.datetime.now(tz=pytz.timezone('UTC'))
        print(dt_now)
        # Works
        gobj = GaugeMetricFamily('FooMetricGood', '')
        gobj.add_metric([], 123, timestamp=dt_now.timestamp())
        yield gobj
        # Works too
        dt_now_str = dt_now.strftime(dt_format)
        dt_parsed = datetime.datetime.strptime(dt_now_str, dt_format)
        gobj = GaugeMetricFamily('FooMetricGoodToo', '')
        gobj.add_metric([], 456, timestamp=dt_parsed.timestamp())
        yield gobj
        # Does not work, but same date
        dt_custom_str = '2021-11-11_18-12-59.000000 +0000'
        dt_parsed_from_custom = datetime.datetime.strptime(dt_custom_str, dt_format)
        gobj = GaugeMetricFamily('FooMetricNotWorking', '')
        gobj.add_metric([], 789987, timestamp=dt_parsed_from_custom.timestamp())
        yield gobj


def main():
    start_http_server(8080)
    REGISTRY.register(BugDemoMetricsCollector())
    while True:
        time.sleep(1)


if __name__ == '__main__':
    sys.exit(main())

Message from Prometheus log about trouble metric:

prometheus-prometheus-1  | ts=2021-11-11T13:41:01.895Z caller=scrape.go:1563 level=warn component="scrape manager" scrape_pool=services target=http://192.168.64.1:8080/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=1

But date is correct, here is Python code for used date and format:

>>> datetime.datetime.strptime('2021-11-11_18-12-59.000000 +0000', '%Y-%m-%d_%H-%M-%S.%f %z')
datetime.datetime(2021, 11, 11, 18, 12, 59, tzinfo=datetime.timezone.utc)

I've spent more than one day trying to google, read docs and resolve it - no results.

Any help will be very appreciated.

Start by reproducing the supplied BugDemoMetricsCollector example and compare the current timestamp with the hard-coded 2021 timestamp in the scrape output. Inspect the client_python timestamp handling and verify the behavior against Prometheus's ingestion warning; done means the reported discrepancy is explained and a regression test covers the relevant case.