cds-snc / cds-snc/notification-planning-core

Celery Logs are not respecting log format on stack traces

Open
#175 0 comments 0 reactions 0 assignees View on GitHub
Bug | Bogue Incident Action Reliability Security | Sécurité Support l Soutien
Dominant language
No language data
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Describe the bug

When celery logs an error with a stack trace, the error is not correctly consumed by the default python multiline parser in fluentbit. All other applications work as expected with the python parser.

## [Bug Severity]

Sev-2

## To Reproduce

- Disable custom celery mutliline parser in fluentbit config
- Redeploy/restart fluentbit to a scratch environment
- Break the signing key in celery to force stack trace errors
- View cloudwatch to see the errors split out over multiple log entries

## Expected behavior

- Celery logs work with the default fluentbit python multiline parser

## Impact

Impact on Notify users: None

Impact on Recipients: None

Impact on Notify team:
Increased maintenance and resource usage on fluentbit in Kubernetes.

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue by disabling the custom Celery multiline parser in the Fluent Bit configuration, restarting Fluent Bit in a scratch environment, and forcing a signing-key failure. Compare the resulting CloudWatch entries with the expected default Python multiline-parser output; done means the stack trace remains one correctly formatted log entry.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.