GAM-team / GAM-team/got-your-back

Rate limitting for backing up & restore to gmail account

Open
#459 0 comments 3 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.1k
Forks
245
PR merge metrics
No merged PRs in 30d

Description

1) To make the gyb.py script work (main branch) I had to insert the sleep() call and use batch size of 5 to pull about 40K messages

def callGAPI(service, function, soft_errors=False, throw_reasons=[], retry_reasons=[], **kwargs):

time.sleep(2) # sleep for X second(s) to avoid hitting the rate limit --batch-size 5 works

this is a hack, should be some parameter.

2) for restore, I got errors with fractional timestamp precision, so I had to chage the below to make this work:

FROM:
```
def convert_timestamp(val):
"""Convert Unix epoch timestamp to datetime.datetime object."""
# yuck, we aren't actually storing timestamps in the GYB
# database. I blame the original developer :-)
# return datetime.datetime.fromtimestamp(int(val))
return datetime.datetime.strptime(val.decode('UTF-8'), '%Y-%m-%d %H:%M:%S')
```

TO:
```
def convert_timestamp(val):
val = val.decode('UTF-8') # Assuming `val` is a bytes object that needs decoding.
# Check if the datetime string contains fractional seconds.
if '.' in val:
# If yes, parse with fractional seconds.
format_str = '%Y-%m-%d %H:%M:%S.%f'
else:
# If no, parse without fractional seconds.
format_str = '%Y-%m-%d %H:%M:%S'
return datetime.datetime.strptime(val, format_str)
```

to handle both values with and without factional values.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in gyb.py by reading callGAPI and convert_timestamp, then trace the backup and restore paths that use them. Reproduce the rate-limit and fractional-timestamp cases described in the issue; done means backup handles the reported Gmail workload without a hard-coded sleep hack and restore accepts timestamps with or without fractional seconds.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.