influxdata / influxdata/influxdb
InfluxDBv2.3 (Ubuntu 22.04) refuses connections after large recurring writing operation (python client)
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
Please let me know how to improve this report or where to place it else if necessary. It's my first of that kind.
__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.
1. Large write from a windows machine with python influxdb-client to an ubuntu server with influxDB v2.3 caused the situation.
__Expected behavior:__
Maybe a few failed writes, e.g. connection losses during writes because of large dataset, but a stable influxdb process in general.
__Actual behavior:__
Many successful writes, several hundred measurements with several million rows each.
More and more errors and connection losses during the writes until writing got stuck completely. I had to kill the script execution manually.
Finally InfluxDB API cannot be reached any more. Everything (writes, queries, GUI) get actively refused.
Rebooting the Ubuntu machine, stopping and starting the service (sudo service influxdb stop/start) did not change anything.
InfluxDB process seems to restart (or be restarted) again and again. Using "sudo service influxdb status" shows the following:
* influxdb.service - InfluxDB is an open-source, distributed, time series datab>
Loaded: loaded (/lib/systemd/system/influxdb.service; enabled; vendor pres>
Active: activating (start) since Mon 2022-09-05 15:44:24 CEST; 1min 17s ago
* Active: activating (start) since Mon 2022-09-05 15:45:55 CEST; 1min 2s ago
* Active: activating (start) since Mon 2022-09-05 15:47:25 CEST; 4s ago
systemctl service status, full file: [systemctl_status_influxdb.txt](https://github.com/influxdata/influxdb/files/9504259/systemctl_status_influxdb.txt)
InfluxDB folder size is 49 GB which is around 10 times larger than expected comparing to other instances witch similar amount of data.
__Environment info:__
* System info: RLinux 5.15.0-39-generic x86_64
* InfluxDB version: InfluxDB v2.3.0+SNAPSHOT.090f681737 (git: 090f681737) build_date: 2022-06-16T19:33:50Z
* client used: python influxdb-client v1.29.1
* client settings: with InfluxDBClient(timeout=120000, verify_ssl=False, enable_gzip=True) as client_v2:
with client_v2.write_api(write_options=WriteOptions(batch_size=5_000, flush_interval=10_000)) as inf2_write_api:
__Config:__
Non-default settings in default config.toml was only TLS activation with selfsigned certificates:
tls-cert = "etc/ssl/influxdb-selfsigned.crt"
tls-key = "etc/ssl/influxdb-selfsigned.key"
__Logs:__
Errors in python prompt during the writes. They were issued more and more often. Some words in German I had to translate. Errors:
* The retriable error occurred during request. Reason: ‘(‘Connection aborted.’, ConnectionResetError(10054, ‘An existing connection was closed by the remote host.’, None, 10054, None))’.
* The retriable error occurred during request. Reason: ‘: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it’.
* The batch item wasn’t processed successfully because: HTTPSConnectionPool(host=‘xxx.xx.xxx.xx’, port=8086): Max retries exceeded with url: /api/v2/write?o…
* Failed to establish a new connection: [WinError 10065] Host could not be reached at a socket process
* Failed to establish a new connection: [WinError 10060] A connection failed, because target did not respond after a certain time, or the established connection was faulty, because the connected host did not respond.'.
__Performance:__
curl profiles:
`curl -o profiles.tar.gz "https://localhost:8086/debug/pprof/all?cpu=30s"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (7) Failed to connect to localhost port 8086 after 0 ms: Connection refused`
curl ping:
`curl -i -X GET http://localhost:8086/ping
curl: (7) Failed to connect to localhost port 8086 after 0 ms: Connection refused`
iostat.txt: [iostat.txt](https://github.com/influxdata/influxdb/files/9491771/iostat.txt)
Contributor guide
Research direction
Start with the attached systemctl_status_influxdb.txt and iostat.txt, then inspect the /api/v2/write, /ping, and /debug/pprof/all endpoints during a large recurring write using the reported Python client settings. Reproduce the service restart and connection refusal on Ubuntu 22.04, and use the service logs and profiles to identify the failure. Done means the cause is isolated and the service remains reachable under the reported workload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100