dandi / dandi/dandisets

minimize storage impact from logging

Open
#123 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
15
Forks
2
PR merge metrics
No merged PRs in 30d

Description

#122 enabled logging into a file at DEBUG level, which could come of great use for troubleshooting in case of errors.

But each log file is about 19MB and in a day we already amassed 0.5GB of them. Clearly it is a bit "wasteful" and we should do something about that. Some ideas

- keep the log only if run errored out and otherwise remove (should probably be possible to setup I bet)
- gzipping folds it to ~10% of its size. May be we should store them gzipped anyways -- why to waste space (although IIRC this partition also has file system level compression enabled)

@jwodder -- wdyt?

Contributor guide

No contributing guide indexed for this repository

Research direction

Review issue #122 and locate the logging configuration that enabled DEBUG-level file logging. Measure the current log growth, then determine how retention or compression should reduce accumulated storage while preserving logs for failed runs; done means the agreed behavior is implemented and verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability-sre
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.