A miniscule number of snapshot measurements are not saved
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 302
- PR merge metrics
- No merged PRs in 30d
Description
In the experiments 2020-05-01-aflplusplus-1 and 2020-05-01-aflplusplus-2 we ran 5 fuzzers on 21 benchmarks with 20 trials for 24 hours with 15 minute snapshot intervals. So we expect there to be 201600 snapshots 5*21*20(24*60/15).
Instead, both experiments have 201599 snapshots.
In 2020-05-01-aflplusplus-1 no snapshot measurement was saved for snapshot 60 of trial 120921.
In 2020-05-01-aflplusplus-2 no snapshot measurement was saved for snapshot 34 of trial 121546.
Since neither of these snapshots is the last, the bug doesn't critically impact results. In the worst case the graph might trend upwards on the next snapshot.
In 2020-05-01-aflplusplus-1 the missing snapshot is an unchanged cycle.
[Here](https://pantheon.corp.google.com/logs/viewer?project=fuzzbench&folder&minLogLevel=0&expandAll=false×tamp=2020-05-04T17:08:41.167000000Z&customFacets=&limitCustomFacetWidth=true&dateRangeEnd=2020-05-04T17:08:37.871Z&interval=PT1H&resource=global&scrollTimestamp=2020-05-04T00:48:50.217464814Z&filters=text:121546&dateRangeUnbound=backwardInTime) are the logs.
Here it looks like it was measured properly but not saved for some reason. I don't really have ideas why this happened.
In 2020-05-01-aflplusplus-1 the missing snapshot is a corpus archive that failed to be copied.
[Here](https://pantheon.corp.google.com/logs/viewer?project=fuzzbench&folder&minLogLevel=0&expandAll=false×tamp=2020-05-04T17:08:41.167000000Z&customFacets=&limitCustomFacetWidth=true&dateRangeEnd=2020-05-04T17:08:37.871Z&interval=PT1H&resource=global&scrollTimestamp=2020-05-04T00:48:50.217464814Z&filters=text:121546&dateRangeUnbound=backwardInTime) are the logs.
The exception is
```
Executed command: "gsutil cp /tmp/work/measurement-folders/bloaty_fuzz_target/aflplusplus_ngram5/trial-121546/crashes-0034.tar.gz gs://fuzzbench-data/2020-05-01-aflplusplus-2/experiment-folders/bloaty_fuzz_target-aflplusplus_ngram5/trial-121546/crashes/crashes-0034.tar.gz" returned: 1."
output: "Traceback (most recent call last):
File "/usr/lib/google-cloud-sdk/platform/gsutil/gsutil", line 21, in
gsutil.RunMain()
File "/usr/lib/google-cloud-sdk/platform/gsutil/gsutil.py", line 124, in RunMain
sys.exit(gslib.__main__.main())
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 433, in main
user_project=user_project)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 762, in _RunNamedCommandAndHandleExceptions
_HandleUnknownFailure(e)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 628, in _RunNamedCommandAndHandleExceptions
user_project=user_project)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 411, in RunNamedCommand
return_code = command_inst.RunCommand()
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/commands/cp.py", line 1162, in RunCommand
copy_helper_opts.daisy_chain,
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/name_expansion.py", line 650, in __init__
name_expansion_dest_tuple = next(self.name_expansion_dest_iter)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/commands/cp.py", line 1093, in _ConstructNameExpansionIteratorDstTupleIterator
logger=self.logger))
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/utils/copy_helper.py", line 1459, in ExpandUrlToSingleBlr
for obj_or_prefix in list_iterator:
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 715, in ListObjects
global_params=global_params)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/third_party/storage_apitools/storage_v1_client.py", line 1217, in List
config, request, global_params=global_params)
File "/usr/lib/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/base_api.py", line 729, in _RunMethod
http, http_request, **opts)
File "/usr/lib/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 356, in MakeRequest
max_retry_wait, total_wait_sec))
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/utils/retry_util.py", line 65, in WarnAfterManyRetriesHandler
http_wrapper.HandleExceptionsAndRebuildHttpConnections(retry_args)
File "/usr/lib/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 304, in HandleExceptionsAndRebuildHttpConnections
raise retry_args.exc
apitools.base.py.exceptions.CommunicationError: Could not reach metadata service: [Errno 111] Connection refused
```
This seems similar to the errors we've seen in the past when running many copies of gsutil at once (I actually don't think this issue is caused by using many instances at once. I think it happens when gsutil is used many times regardless of them being done in parallel or sequentially). Any thoughts on this @inferno-chromium? I think this is probably not worth fixing as we will change the measurer significantly in the near future.
Contributor guide
Assessment
This issue has not been assessed yet.