fullstorydev / fullstorydev/hauser

Download times out while decoding JSON

Open
#82 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
50
Forks
21
PR merge metrics
No merged PRs in 30d

Description

I am getting repeated error messages like:
```
16:20:34 Processing bundle 156453120048 (start: 2019-07-31 00:00:00 +0000 UTC, end: 2019-08-01 00:00:00 +0000 UTC)
16:20:34 Getting Export Data for bundle 156453120048
16:30:36 failed json decode of record: unexpected EOF
16:30:43 Pausing; will retry operation in 30s
16:31:13 Checking if table fs_sync exists
16:31:14 Checking if table fs_export exists
16:31:15 Checking for new export files since 2019-07-31 00:00:00 +0000 UTC
16:31:15 Processing bundle 156453120048 (start: 2019-07-31 00:00:00 +0000 UTC, end: 2019-08-01 00:00:00 +0000 UTC)
...
```

You can see the time from `Getting Export Data` to `failed json decode` is about 10 minutes, which is consistent through many loops of this error. I think this is a timeout for DataExport downloads from the REST api. Our larger export files reach many GB in size, so we're hitting this problem running hauser on our slower VMs.

I tried manually skipping a couple DataExport files to see if it was only one specific file having the problem, but I hit the same problem for the next two days of DataExport files.

My config.toml
```
FsApiToken = "redacted"
Backoff = "30s"
BackoffStepsMax = 8
CheckInterval = "30m"
TmpDir = "/tmp"
Warehouse="bigquery"
GroupFilesByDay = false
SaveAsJson = false

[gcs]
Bucket = "redacted"
GCSOnly = false

[bigquery]
Project = "redacted"
Dataset = "redacted"
ExportTable = "fs_export"
SyncTable = "fs_sync"
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.