influxdata / influxdata/influxdb

InfluxDBv2.3 (Ubuntu 22.04) refuses connections after large recurring writing operation (python client)

Open
#23,712 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

Please let me know how to improve this report or where to place it else if necessary. It's my first of that kind.

__Steps to reproduce:__

List the minimal actions needed to reproduce the behavior.

1. Large write from a windows machine with python influxdb-client to an ubuntu server with influxDB v2.3 caused the situation.

__Expected behavior:__
Maybe a few failed writes, e.g. connection losses during writes because of large dataset, but a stable influxdb process in general.

__Actual behavior:__
Many successful writes, several hundred measurements with several million rows each.
More and more errors and connection losses during the writes until writing got stuck completely. I had to kill the script execution manually.
Finally InfluxDB API cannot be reached any more. Everything (writes, queries, GUI) get actively refused.
Rebooting the Ubuntu machine, stopping and starting the service (sudo service influxdb stop/start) did not change anything.
InfluxDB process seems to restart (or be restarted) again and again. Using "sudo service influxdb status" shows the following:

* influxdb.service - InfluxDB is an open-source, distributed, time series datab>
Loaded: loaded (/lib/systemd/system/influxdb.service; enabled; vendor pres>
Active: activating (start) since Mon 2022-09-05 15:44:24 CEST; 1min 17s ago
* Active: activating (start) since Mon 2022-09-05 15:45:55 CEST; 1min 2s ago
* Active: activating (start) since Mon 2022-09-05 15:47:25 CEST; 4s ago

systemctl service status, full file: [systemctl_status_influxdb.txt](https://github.com/influxdata/influxdb/files/9504259/systemctl_status_influxdb.txt)

InfluxDB folder size is 49 GB which is around 10 times larger than expected comparing to other instances witch similar amount of data.

__Environment info:__

* System info: RLinux 5.15.0-39-generic x86_64
* InfluxDB version: InfluxDB v2.3.0+SNAPSHOT.090f681737 (git: 090f681737) build_date: 2022-06-16T19:33:50Z
* client used: python influxdb-client v1.29.1
* client settings: with InfluxDBClient(timeout=120000, verify_ssl=False, enable_gzip=True) as client_v2:
with client_v2.write_api(write_options=WriteOptions(batch_size=5_000, flush_interval=10_000)) as inf2_write_api:

__Config:__
Non-default settings in default config.toml was only TLS activation with selfsigned certificates:
tls-cert = "etc/ssl/influxdb-selfsigned.crt"
tls-key = "etc/ssl/influxdb-selfsigned.key"

__Logs:__
Errors in python prompt during the writes. They were issued more and more often. Some words in German I had to translate. Errors:

* The retriable error occurred during request. Reason: ‘(‘Connection aborted.’, ConnectionResetError(10054, ‘An existing connection was closed by the remote host.’, None, 10054, None))’.
* The retriable error occurred during request. Reason: ‘: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it’.
* The batch item wasn’t processed successfully because: HTTPSConnectionPool(host=‘xxx.xx.xxx.xx’, port=8086): Max retries exceeded with url: /api/v2/write?o…
* Failed to establish a new connection: [WinError 10065] Host could not be reached at a socket process
* Failed to establish a new connection: [WinError 10060] A connection failed, because target did not respond after a certain time, or the established connection was faulty, because the connected host did not respond.'.

__Performance:__
curl profiles:
`curl -o profiles.tar.gz "https://localhost:8086/debug/pprof/all?cpu=30s"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (7) Failed to connect to localhost port 8086 after 0 ms: Connection refused`

curl ping:
`curl -i -X GET http://localhost:8086/ping
curl: (7) Failed to connect to localhost port 8086 after 0 ms: Connection refused`

iostat.txt: [iostat.txt](https://github.com/influxdata/influxdb/files/9491771/iostat.txt)

Contributor guide

Open the contributing guide

Research direction

Start with the attached systemctl_status_influxdb.txt and iostat.txt, then inspect the /api/v2/write, /ping, and /debug/pprof/all endpoints during a large recurring write using the reported Python client settings. Reproduce the service restart and connection refusal on Ubuntu 22.04, and use the service logs and profiles to identify the failure. Done means the cause is isolated and the service remains reachable under the reported workload.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.