Error on ingesting out-of-order samples

for the last 5+ days, I'm getting via Prometheus the error

component="scrape manager" scrape_pool=serviceMonitor/monitoring/prometheus-stackdriver-exporter/0 target=http://10.205.6.116:9255/metrics msg="Error on ingesting out-of-order samples" num_dropped=946 

I tried playing with the offsets and intervals, but I cannot fix this error,
the error I am getting in alertmanager is PrometheusDuplicateTimestamps
and I'm using chart version 4.4.1 (I upgraded it after the alert started showing up)

here are the values I pass to configure this exporter

stackdriver:
  projectId: '{{values.projectId}}'
  metrics:
    typePrefixes: >-
      logging.googleapis.com/user,
      pubsub.googleapis.com,
      bigtable.googleapis.com,
      cloudsql.googleapis.com,
      custom.googleapis.com,
      dataflow.googleapis.com,
      monitoring.googleapis.com/uptime_check,
      serviceruntime.googleapis.com/quota,
      file.googleapis.com,
      loadbalancing.googleapis.com
resources:
  requests:
    cpu: 100m
    memory: 128Mi
serviceMonitor:
  enabled: true
  interval: 20s
  additionalLabels:
    release: prometheus

and some time I get this in the logs

ts=2023-12-27T17:13:05.253Z caller=monitoring_collector.go:315 level=error msg="error retrieving Time Series metrics for descriptor" descriptor=cloudsql.googleapis.com/database/mysql/innodb_buffer_pool_pages_dirty err="Get \"https://monit
oring.googleapis.com/v3/projects/fakeproject/timeSeries?alt=json&filter=metric.type%3D%22cloudsql.googleapis.com%2Fdatabase%2Fmysql%2Finnodb_buffer_pool_pages_dirty%22&interval.endTime=2023-12-27T17%3A13%3A05.233788804Z&interval.startTime=20
23-12-27T17%3A08%3A05.233788804Z&prettyPrint=false\": EOF"
ts=2023-12-27T17:13:05.253Z caller=monitoring_collector.go:315 level=error msg="error retrieving Time Series metrics for descriptor" descriptor=cloudsql.googleapis.com/database/mysql/open_table_definitions err="Get \"https://monitoring.go
ogleapis.com/v3/projects/fakeproject/timeSeries?alt=json&filter=metric.type%3D%22cloudsql.googleapis.com%2Fdatabase%2Fmysql%2Fopen_table_definitions%22&interval.endTime=2023-12-27T17%3A13%3A05.233788804Z&interval.startTime=2023-12-27T17%3A08
%3A05.233788804Z&prettyPrint=false\": EOF"
ts=2023-12-27T17:13:05.253Z caller=monitoring_collector.go:315 level=error msg="error retrieving Time Series metrics for descriptor" descriptor=cloudsql.googleapis.com/database/sqlserver/transactions/forwarded_record_count err="Get \"http
s://monitoring.googleapis.com/v3/projects/fakeproject/timeSeries?alt=json&filter=metric.type%3D%22cloudsql.googleapis.com%2Fdatabase%2Fsqlserver%2Ftransactions%2Fforwarded_record_count%22&interval.endTime=2023-12-27T17%3A13%3A05.233788804Z&i
nterval.startTime=2023-12-27T17%3A08%3A05.233788804Z&prettyPrint=false\": EOF"
ts=2023-12-27T17:13:05.254Z caller=monitoring_collector.go:315 level=error msg="error retrieving Time Series metrics for descriptor" descriptor=cloudsql.googleapis.com/database/replication/log_archive_failure_count err="Get \"https://moni
toring.googleapis.com/v3/projects/fakeproject/timeSeries?alt=json&filter=metric.type%3D%22cloudsql.googleapis.com%2Fdatabase%2Freplication%2Flog_archive_failure_count%22&interval.endTime=2023-12-27T17%3A13%3A05.233788804Z&interval.startTime=
2023-12-27T17%3A08%3A05.233788804Z&prettyPrint=false\": EOF"
ts=2023-12-27T17:13:05.254Z caller=monitoring_collector.go:315 level=error msg="error retrieving Time Series metrics for descriptor" descriptor=cloudsql.googleapis.com/database/postgresql/insights/perquery/lock_time err="Get \"https://mon
itoring.googleapis.com/v3/projects/fakeproject/timeSeries?alt=json&filter=metric.type%3D%22cloudsql.googleapis.com%2Fdatabase%2Fpostgresql%2Finsights%2Fperquery%2Flock_time%22&interval.endTime=2023-12-27T17%3A13%3A05.233788804Z&interval.star
tTime=2023-12-27T17%3A08%3A05.233788804Z&prettyPrint=false\": EOF"
ts=2023-12-27T17:13:05.693Z caller=monitoring_collector.go:228 level=error msg="Error while getting Google Stackdriver Monitoring metrics" err="Get \"https://monitoring.googleapis.com/v3/projects/fakeproject/timeSeries?alt=json&filter=metric
.type%3D%22cloudsql.googleapis.com%2Fdatabase%2Fmysql%2Finnodb_buffer_pool_pages_dirty%22&interval.endTime=2023-12-27T17%3A13%3A05.233788804Z&interval.startTime=2023-12-27T17%3A08%3A05.233788804Z&prettyPrint=false\": EOF"

Start with monitoring_collector.go around lines 228 and 315, then inspect how the exporter retrieves Google Cloud time series and exposes them to Prometheus. Reproduce the out-of-order and duplicate-timestamp errors using the supplied metric configuration and determine whether the EOF responses contribute. Done means the cause is identified and the exporter no longer emits invalid samples under the affected conditions, with regression coverage if an existing test location is found.