jorgebastida / jorgebastida/awslogs

Missing data

Open
#342 4 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
5k
Forks
342
PR merge metrics
No merged PRs in 30d

Description

I was using AwsLogs to request a large volume of data over an extended period and when I repeated the exact same request the resulting logfiles were different. I used something along the lines of the following command:
awslogs get -G --timestamp _log-group_ "_log-stream_-*" --filter-pattern="_node-id_." -s "2021-02-19 11:00" -e "2021-03-08 11:59:59" | grep "_node-id_.[RT]X" > "_node-id_.log"

In one instance all data from 2021-02-26 till 2021-03-02 was missing from one of the logging steps, in another case only 5 minutes of data was missing.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the issue with the reported awslogs get command over the same date range, stream pattern, and filter pattern, comparing repeated outputs. Trace the command's CloudWatch log retrieval path and identify where records may be skipped; done means the missing-data behavior is explained and covered by a regression test or a documented reproduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cli, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.