[Bug] Postgres data using too much disk space and doesn't cleaning automatic
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 22.6k
- Forks
- 1.5k
- Avg merge
- 4h 30m
- Merged PRs (30d)
- 11
Description
What happened?
Hi. I`m using safeline CE docker installation version 8.4.0 and my postgres data using too much disk space
I have 5 active domains and ~1.5m-2m requests per day
I set keeping logs for 1 day in safeline UI. But it didn't help, my pg data using 28 GB on disk.
How we reproduce?
- Set clean data 1 day in web UI
- Checking postgres tables size
- All tables are cleaning, except pg_toast_16416
Expected behavior
Postgres using less disk space
Error log
My settings:
I also ran SQL query:
SELECT relname, relpages
FROM pg_class
ORDER BY relpages DESC;
and got this results:
relname | relpages
------------------------------------------------+----------
pg_toast_16416 | 2439830
mgt_detect_log_detail | 617415
mgt_detect_log_basic | 209362
idx_mgt_detect_log_detail_event_id | 50358
idx_mgt_detect_log_basic_event_id | 50123
pg_toast_16416_index | 33296
mgt_detect_log_detail_pkey | 14915
mgt_detect_log_basic_pkey | 14849
idx_detect_log_basic_src_ip_host_dst_port | 6541
idx_mgt_detect_log_basic_socket_ip | 6272
idx_mgt_detect_log_detail_timestamp | 5306
idx_mgt_detect_log_basic_created_at | 5284
idx_mgt_detect_log_basic_timestamp | 5283
idx_mgt_detect_log_basic_action | 4890
idx_mgt_detect_log_basic_attack_type | 4810
idx_mgt_detect_log_basic_risk_level | 4805
mgt_event | 1681
statistics_ips | 1158
resources | 816
statistics_ips_pkey | 414
mgt_statistics_agg | 405
mgt_event_pkey | 337
idx_statistics_ips_key | 269
pg_toast_16696 | 218
idx_mgt_event_ip | 210
idx_statistics_ips_day_time | 159
statistics_basics | 146
idx_statistics_ips_time | 141
idx_mgt_event_finished | 99
statistics_geos_pkey | 98
pg_attribute | 96
pg_proc | 96
idx_statistics_geo_time_country | 91
idx_mgt_event_host | 91
idx_mgt_event_dst_port | 84
qps | 78
statistics_geos | 73
mgt_statistics_agg_pkey | 67
pg_toast_2618 | 64
statistics_basics_pkey | 55
pg_statistic | 55
statistics_sessions | 54
pg_toast_2619 | 52
siteid_path_method | 45
pg_description | 44
idx_statistics_agg_type_time | 37
idx_resources_path | 34
qps_pkey | 33
pg_proc_proname_args_nsp_index | 31
idx_statistics_sessions_key | 30
pg_description_o_c_o_index | 28
idx_statistics_basic_type_time | 27
idx_statistics_basics_site_id | 27
statistics_sessions_pkey | 27
idx_mgt_statistics_agg_day_time | 25
idx_mgt_statistics_agg_hour_time | 23
pg_attribute_relid_attnam_index | 23
pg_depend | 20
table pg_toast_16416 is very big and seem like it doesn't cleaning at all
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the one-day retention setting from the SafeLine UI to cleanup for mgt_detect_log_basic and mgt_detect_log_detail, using the supplied pg_class query to inspect pg_toast_16416. Done means the reported large PostgreSQL storage is addressed after retention cleanup and the behavior is covered by an appropriate regression check.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, postgresql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100