GAM-team / GAM-team/got-your-back
Rate limitting for backing up & restore to gmail account
- Dominant language
- Python
- Stars
- 3.1k
- Forks
- 245
- PR merge metrics
- No merged PRs in 30d
Description
1) To make the gyb.py script work (main branch) I had to insert the sleep() call and use batch size of 5 to pull about 40K messages
def callGAPI(service, function, soft_errors=False, throw_reasons=[], retry_reasons=[], **kwargs):
time.sleep(2) # sleep for X second(s) to avoid hitting the rate limit --batch-size 5 works
this is a hack, should be some parameter.
2) for restore, I got errors with fractional timestamp precision, so I had to chage the below to make this work:
FROM:
```
def convert_timestamp(val):
"""Convert Unix epoch timestamp to datetime.datetime object."""
# yuck, we aren't actually storing timestamps in the GYB
# database. I blame the original developer :-)
# return datetime.datetime.fromtimestamp(int(val))
return datetime.datetime.strptime(val.decode('UTF-8'), '%Y-%m-%d %H:%M:%S')
```
TO:
```
def convert_timestamp(val):
val = val.decode('UTF-8') # Assuming `val` is a bytes object that needs decoding.
# Check if the datetime string contains fractional seconds.
if '.' in val:
# If yes, parse with fractional seconds.
format_str = '%Y-%m-%d %H:%M:%S.%f'
else:
# If no, parse without fractional seconds.
format_str = '%Y-%m-%d %H:%M:%S'
return datetime.datetime.strptime(val, format_str)
```
to handle both values with and without factional values.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in gyb.py by reading callGAPI and convert_timestamp, then trace the backup and restore paths that use them. Reproduce the rate-limit and fractional-timestamp cases described in the issue; done means backup handles the reported Gmail workload without a hard-coded sleep hack and restore accepts timestamps with or without fractional seconds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100