Download failures when network is saturated

The Issue

When @shyba and I looked into some recent download failures, even on blob mirroring, they were happening because the user was already downloading another file that was saturating their connection. This would cause DNS errors in the logs as well as sdblob/datablob timeouts. Part of the solution would be https://github.com/lbryio/lbry/issues/798, but we'd also need a way to automatically set the max. We could also allow certain calls (initial get/wallet commands) to have priority over existing download calls. More exploration is needed to see what can be done to prevent this.

If you'd like to see some more logs, shoot me a DM!

2019-01-12 21:48:49,263 INFO     lbrynet.p2p.HTTPBlobDownloader:143: Mirror download cancelled: a98c172b
fbf5d26e138cbd4f42f4542e0ae9a70701940d0e84bfc3635c8c6a769cf750ff9654ebc0c8c32147f7332c67
2019-01-12 21:48:58,970 WARNING  lbrynet.extras.daemon.Components:707: upnp discovery failed: M-SEARCH f
or 192.168.42.129:1900 timed out
2019-01-12 21:48:58,970 WARNING  lbrynet.extras.daemon.Components:721: unable to get external ip from UP
nP, checking lbry.io fallback
2019-01-12 21:49:01,384 INFO     lbrynet.p2p.HTTPBlobDownloader:132: Mirror completed download for 84523
26315b99d7ba5ebc7a1307f6011a9e7db5bd5b08be07648a1a5fa8a9e5517c737f68a538e911a2570439a3e116b
2019-01-12 21:49:08,130 ERROR    lbrynet.p2p.HTTPBlobDownloader:146: Mirror failed downloading
Traceback (most recent call last):
  File "lbrynet\p2p\HTTPBlobDownloader.py", line 128, in _download_blob
twisted.web._newclient.ResponseFailed: [<twisted.python.failure.Failure twisted.internet.error.Connectio
nLost: Connection to the other side was lost in a non-clean fashion.>, <twisted.python.failure.Failure t
wisted.web.http._DataLoss: >]
2019-01-12 21:49:08,727 INFO     lbrynet.p2p.client.BlobRequester:564: Blob c4e40d91d585f8a0 has been su
ccessfully downloaded from 85.17.24.157:4444
2019-01-12 21:49:08,731 INFO     lbrynet.p2p.client.BlobRequester:488: Requesting blob 1d3ec0b2880ddc781
1c37dd9c03a1f11baab368cf16f4fd4698ca24ea361fabc3f544e009dc9cea7e8ff1f82cad89ee0 from 85.17.24.157:4444
2019-01-12 21:49:17,321 INFO     lbrynet.p2p.HTTPBlobDownloader:132: Mirror completed download for 0298e
59d3f38fd8d1931d0351f15e98e32ccb211f038e693be7f82af0f32365db03d8ae806e7589318b242ca17ad0326
2019-01-12 21:49:17,321 INFO     lbrynet.p2p.client.ClientProtocol:182: Closing the connection to 24.184
.108.248:3334 because the download of blob 0298e59d3f38fd8d was canceled
2019-01-12 21:49:17,322 INFO     lbrynet.p2p.client.BlobRequester:488: Requesting blob 9e3671b3496ea3f1419bc91c64be78dd18ceb39e099024819885bb388b25792d5a89c6c8c3419b7c92fcf1d67a3f2a6c from 24.184.108.248:3334
2019-01-12 21:49:17,323 INFO     lbrynet.p2p.client.ClientProtocol:182: Closing the connection to 24.184.108.248:3334 because the download of blob 9e3671b3496ea3f1 was canceled
2019-01-12 21:49:23,626 INFO     lbrynet.p2p.client.BlobRequester:415: Offered rate 0.000000/mb accepted by 24.184.108.248
2019-01-12 21:49:23,627 INFO     lbrynet.p2p.client.BlobRequester:488: Requesting blob 8e61c92cb30751b59b9e45fa817fbd65d4f3f376f70cd2dd3890dd9c30e48b6620050712d5b114c0f9302dcd996b760d from 24.184.108.248:3334
2019-01-12 21:49:33,022 INFO     lbrynet.p2p.HTTPBlobDownloader:132: Mirror completed download for a403bf3288cb6ba2df64a7ef45f079ec69265a45c4b9a2c3f580f058496733f9190e4a4f929eea9ce3e62ff0ce40c5ee
2019-01-12 21:49:33,693 CRITICAL twisted:154: Unhandled error in Deferred:
2019-01-12 21:49:33,693 CRITICAL twisted:154: 
Traceback (most recent call last):
  File "lbrynet\extras\daemon\Downloader.py", line 193, in start
  File "twisted\internet\defer.py", line 654, in _runCallbacks
  File "twisted\internet\defer.py", line 395, in convertCancelled
  File "twisted\internet\defer.py", line 863, in _cancelledToTimedOutError
twisted.internet.defer.TimeoutError: (180, 'Deferred')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "twisted\internet\defer.py", line 654, in _runCallbacks
  File "lbrynet\extras\daemon\Downloader.py", line 135, in fail
  File "lbrynet\extras\daemon\Daemon.py", line 376, in _download_name
  File "twisted\internet\defer.py", line 1416, in _inlineCallbacks
  File "twisted\python\failure.py", line 491, in throwExceptionIntoGenerator
  File "lbrynet\extras\daemon\Downloader.py", line 198, in start
  File "lbrynet\extras\daemon\Downloader.py", line 196, in start
lbrynet.p2p.Error.DownloadDataTimeout: Failed to download data blobs for sd hash data download timed out within timeout

System Configuration

• LBRY Daemon version:
• LBRY App version:
• LBRY Installation ID:
• Operating system:

Anything Else

Screenshots

Internal Use

Acceptance Criteria

Definition of Done

• Tested against acceptance criteria
• Tested against the assumptions of user story
• The project builds without errors
• Unit tests are written and passing
• Tests on devices/browsers listed in the issue have passed
• QA performed & issues resolved
• Refactoring completed
• Any configuration or build changes documented
• Documentation updated
• Peer Code Review performed

Start by reviewing the stack-trace entry points in lbrynet/p2p/HTTPBlobDownloader.py, lbrynet/extras/daemon/Downloader.py, and lbrynet/extras/daemon/Daemon.py, along with referenced issue #798. Investigate how saturated connections produce DNS errors and blob timeouts. Done requires a defined approach for automatic download limits or request priority, with unit tests and verification against the reported failure.