improve sync / conversion behavior on poor connections/disconnect (balance issues)

The Issue

I recently helped out a user that was trying to convert an old large wallet (many transactions at a single address) with a poor internet connection. We need to better suppor the sync process during failures, including connection and claim decoding.

If the connection timed out/failed, the wallet would fail to load history and then set incorrect gaps.

History on https://explorer.lbry.io/address/bK35BP1PxY47KqcSqgtRiRXkjwsrSfNP37

2018-12-17 23:54:38,370 INFO     torba.client.baseledger:150: (lbc_mainnet) on_address: ['bY6zGt7QDGBfePEKV3KaQTRh2wabJWP7v8', 'bKXAiLML1gZeHzTrM1M1ApQtRCFkCLMjZt', 'bJrLonap8QiFVpJHHZwcKFrKdu1muQcr9k', 'bSpUA2vzcdrwNzrpdncKkPHFVpJtJNqngk', 'bQd6WAiSUdJN4qg7UABjbciiNePduoCAf4', 'bKE9Rw8tJRwDpsYX9u7sAPsDbeAaes5jvo', 'bCxMddPzPsAtVU3xrVuwwrNHEEr8dgGUMk', 'bDcSqzKWKQRtwTSYMH1ddYo32M7grgKMe6', 'bE585fNc49wLnHvNr2WqaJbhJw9CVGV9bb', 'bbBPgbpdAGgLbQrP1n3BE16azNJTACPQX5', 'bQKokGDGNM4qRAhkF8CN8JnVE2pfa12yKk', 'bMWczy4QQmZpHoQkxhkVC43vA3PBgxRsYi', 'bZapNZoemycQDVB1KQSUB33yYXER5JUYRS', 'bGEiab8ES3ZZstZXDvyBoarmSDvncGBnUf', 'bM9cgCKv6cMzNaR8iu9U7fUZK4vpNBc31x', 'bRhs6mmfvGNSgx6DYK5zGi1ForpRNa1E88', 'bPG5ZugnfLvsVaMHZkffzwbfDwhH18mFEG', 'bNgU7jFAQUMTFvXb8WN15XYJQY8DwgmKSg', 'bUpTfv9dDsQ9ipDmPakjKRB8roBMiNntA2', 'bUVExcRim7eMrxPjbrLa9zbe2FSs6hLyiC', 'bbsMj2bnMCbw92xRLLa1CYD1FMqTefq2a2', 'bXSiggXrh6UeeVm6TjfiuZtFrfpCpyxm3t', 'bK35BP1PxY47KqcSqgtRiRXkjwsrSfNP37', 'bUkZDxqCmgYQEadTthC66JZpy3KjGDW2Py', 'bZUvVD8b4C2ivNVTYvGyu5Hrz4oK4t5NFQ', 'bQFd5kcvsTHbgvyF6hDygwUgJ4Dk9gDXJ8', 'bFGXGikogVDYU3Auytpj4HzQVK3p7c8iHa', 'bXd4WVZBHUoEBEbnpdxJFHZg2mSePS33EB', 'bYtwS9TjXZjU3nghJUCeEgLRHUeAWiF29P', 'bDydcbv2n82gM74eZmtsKRgBob3rXhjdUQ', 'bYLdqTtJtsUMNMhr4TjoVaFaEdXJbF8AyX', 'bP16rn6xzqH2icjjkUgU6xCjaMnTs1SKWP', 'bEQqXFy6CVaA8YM7hkoaahv2W2TmW2oXAa', 'bZTPHrfYKNA75vsfj44742DRAxdTSa4nS4', 'bLBrdGWGxwVfD5per6eUhkbuJC4aUxPEap', 'bJCxLjEX3S2Q3WvszNFnFNaD6pY7bTkD9K', 'bJikdEgki1WAV2DBHNUPTJceGWfkiiMU5N', 'bTyEr7YrpSjoV7ibAtmdt9Rb8BDfAZodWx', 'bQSkZ3ZDdoYhkbiz3z68Spw6JLEjs2vGEi', 'bbwfqhDduCY7N2Sm9WiMAw4pEBCAWEXSp8', 'bVe6mSL4cKNUgYiB4tPDdmBCRYQgbdAoGL', 'bFnDQtGDAnraSQMKZLgQvB1WfDg79skrt9', 'bXoPBJcW2knPed4YXtBfuj53Y7ptBDqFhJ', 'bQt9gZJtadkkJzSsLU34zk3XyvZ8E5k7QG', 'bX3xRfWedR7h6iho55vBWTEVmHoCJTMBtk', 'bMHDz8YWAjXiKq24HfXpoEDbxteuaTCSCX', 'bN7mjYecGcPUyG83yGTgupJ6y3Zgnmnkw7', 'bL1qVwd2yCpwY3M9Na1CEVQqvk9HuAFoU1', 'bN4ZEWGgQFMAAhThahuc9qVmU2V5EbdU8T', 'bMCMU64KyagAS4WnziSVgmqXfkX51GAbid']
2018-12-17 23:54:47,218 INFO     lbrynet.dht.node:227: Attempting to join the DHT network, 8 contacts known so far
2018-12-17 23:54:53,341 INFO     torba.client.baseledger:150: (lbc_mainnet) on_address: ['bYr8AtZXc9xLBbCun9C9wyBWoB9CYS9xrB', 'bRyVrVoxWXHimF6SFPeCqh3SKyjGcEcL1J', 'bRjrYoMm4FbQKDLe9wMoZzWkxogSGiQvGX', 'bRYhjb6Rsp6jNazqKvEninXzcUokNiATQW', 'bGPzjgvpHzk3kc5oSNmfg6rZPoNSPUYbYr', 'bD4cbyFgZxF1PDmPkF2fofNvUTPF37GNRo', 'bMPXx5fF1p54XiqyjZ9jnkY6UqqgKHPotz', 'bM6ppmrS37h14twrHT5yogbRgSrArPAZ1Z', 'bX9w45hiQi6Z1dfFsfwhCrBGtGKK4DP5Uq', 'bZFGnmDSyhoejMS71VXk25M4kNvAup6mJb', 'bMQNfqpic8xC3Fn35tBn9468fq1qtijZqq', 'bbQdHRNBgUNUWdiktwUSuiUEmfaojwLkRn', 'bQrK562xTozXQQPHW8nUfJBtV9LZfQRq1X', 'bbnjGfxSguQm7o6es3LNJU4YrEjBZ3GF7P', 'bYfAMhUsueZZNebA9aYevWj1NCnKrftpEA', 'baSUDdycvyBxptYEkXpWqCv6TZqufA5x2G', 'bUAXj7EUrqdwP1YgexQBo45zrGWc4Kb7SU', 'bZxDLcoPymwcN2HHPNYQ8tgXR1FNc8rgse', 'bZ8TsjVdHtKGU2zQTA1ytdfVzp1Pgqbesx', 'bYwNaqnUEjbZm2kgyjxZGKErMyCjrrtv4n']
2018-12-17 23:55:07,003 INFO     lbrynet.dht.node:263: Joined the dht
2018-12-17 23:55:56,232 INFO     lbrynet.extras.wallet.ledger:100: Loaded account bYhqPPHEBXBJdwH6rHUatrazctsaCSBUHk with 0.0 LBC, 50 receiving addresses (gap: 20), 20 change addresses (gap: 6), 0 channels, 0 certificates and 0 claims. 

Sometimes he'd run into this:

Traceback (most recent call last):
  File "lbrynet\extras\daemon\Component.py", line 55, in _setup
  File "twisted\internet\defer.py", line 1416, in _inlineCallbacks
  File "twisted\python\failure.py", line 491, in throwExceptionIntoGenerator
  File "lbrynet\extras\daemon\Components.py", line 369, in start
  File "twisted\internet\defer.py", line 824, in adapt
  File "lbrynet\extras\wallet\manager.py", line 201, in from_lbrynet_config
  File "lbrynet\extras\wallet\manager.py", line 208, in _migrate_addresses
  File "torba\client\baseaccount.py", line 142, in _generate_keys
  File "torba\client\basedatabase.py", line 501, in add_keys
  File "concurrent\futures\thread.py", line 57, in run
  File "torba\client\basedatabase.py", line 57, in __run_transaction
  File "torba\client\basedatabase.py", line 49, in <lambda>
sqlite3.IntegrityError: UNIQUE constraint failed: pubkey_address.address

System Configuration

• LBRY Daemon version:
• LBRY App version:
• LBRY Installation ID:
• Operating system:

Anything Else

Screenshots

Internal Use

Acceptance Criteria

Definition of Done

• Tested against acceptance criteria
• Tested against the assumptions of user story
• The project builds without errors
• Unit tests are written and passing
• Tests on devices/browsers listed in the issue have passed
• QA performed & issues resolved
• Refactoring completed
• Any configuration or build changes documented
• Documentation updated
• Peer Code Review performed

Start with lbrynet/extras/wallet/manager.py at _migrate_addresses, then trace the reported calls through torba/client/baseaccount.py (_generate_keys) and torba/client/basedatabase.py (add_keys and transaction handling). Reproduce wallet conversion or loading with a poor connection and inspect how failed sync and claim decoding affect gaps. Done means failures do not leave incorrect gaps or trigger the reported SQLite duplicate-address error.