GoogleCloudPlatform / GoogleCloudPlatform/PerfKitBenchmarker

Downloading packages via curl is unstable.

Open
#686 2 comments 0 reactions 0 assignees View on GitHub
enhancement P1
Dominant language
Python
Stars
2k
Forks
562
Avg merge
4h 55m
Merged PRs (30d)
69

Description

`hadoop_terasort` benchmark during prepare stage downloads Hadoop archive via curl. It's a pretty big package (141M) and many times I have encountered an issue with a broken connection like this:

```
RemoteCommandError: Got non-zero return code (2) executing mkdir /tmp/pkb/hadoop && curl -L http://www.us.apache.org/dist/hadoop/common/hadoop-2.5.2/hadoop-2.5.2.tar.gz | tar -C /tmp/pkb/hadoop --strip-components=1 -xzf -
Full command: ssh -A -p 22359 root@kp37.intelsdi.com -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /tmp/perfkitbenchmarker/run_9d08414d/perfkitbenchmarker_keyfile mkdir /tmp/pkb/hadoop && curl -L http://www.us.apache.org/dist/hadoop/common/hadoop-2.5.2/hadoop-2.5.2.tar.gz | tar -C /tmp/pkb/hadoop --strip-components=1 -xzf -
STDOUT: STDERR: Warning: Permanently added '[kp37.intelsdi.com]:22359,[10.4.1.37]:22359' (ECDSA) to the list of known hosts.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
1 140M 1 1921k 0 0 527k 0 0:04:32 0:00:03 0:04:29 527k
curl: (18) transfer closed with 145229430 bytes remaining to read

gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
```

It causes the benchmark to fail. I may try to re-run the benchmark and I've got like 15% of changes that archives in all the instances will be downloaded correctly. I would propose to switching to `wget` instead of `curl`. I have checked that it automatically retries the download process:

```
cd /tmp ; wget http://www.us.apache.org/dist/hadoop/common/hadoop-2.5.2/hadoop-2.5.2.tar.gz && tar -C /tmp/pkb/hadoop --strip-components=1 -xzf hadoop-2.5.2.tar.gz
--2015-12-01 10:24:16-- http://www.us.apache.org/dist/hadoop/common/hadoop-2.5.2/hadoop-2.5.2.tar.gz
Resolving www.us.apache.org (www.us.apache.org)... 140.211.11.105
Connecting to www.us.apache.org (www.us.apache.org)|140.211.11.105|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 147197492 (140M) [application/x-gzip]
Saving to: 'hadoop-2.5.2.tar.gz'

16% [==================> ] 24,045,195 299KB/s eta 7m 16s
50% [==========================================================> ] 74,792,595 141KB/s in 5m 0s

2015-12-01 10:29:17 (243 KB/s) - Connection closed at byte 74792595. Retrying.

--2015-12-01 10:29:18-- (try: 2) http://www.us.apache.org/dist/hadoop/common/hadoop-2.5.2/hadoop-2.5.2.tar.gz
Connecting to www.us.apache.org (www.us.apache.org)|140.211.11.105|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 147197492 (140M), 72404897 (69M) remaining [application/x-gzip]
Saving to: 'hadoop-2.5.2.tar.gz'

58% [+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++========> ] 86,134,101 211KB/s eta 3m 31s
```

You can see that after 50% the connection was broken but it was automatically retried.
Another option is described here:
http://ngtuna.blogspot.de/2013/12/download-continuously-with-curl.html
But personally I think switching to wget would be simpler.

This issue concerns probably more benchmarks than only `hadoop_terasort`. Potentially any benchmark using curl may encounter the same issue.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.