chaoss / chaoss/CollectOSS

Single log line is 30+ MB in size

Open
#401 0 comments 0 reactions 0 assignees View on GitHub
devops
Dominant language
Python
Stars
13
Forks
17
Avg merge
6h 59m
Merged PRs (30d)
1

Description

When debugging other things, I took a dump of the docker logs and discovered there to be a single line that, when saved to its own file, was 32.6 MiB in size.

This is insane.

this line came from postgres logging a query line (unsure whether it was an error or not because it was truncated)

This query was attempting to insert (at least from what i could see) at least 85422 records into the repo_labor table.

This logging is probably also made worse by the recursive way we process data in bulk_insert_dicts (causing failures to show up many times).

the affected line
https://github.com/chaoss/CollectOSS/blob/d289de1b69c3cc1640ff53b5401a03a9972c5af6/collectoss/tasks/git/scc_value_tasks/core.py#L53

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.