Improve reconciliation service error handling

With the wonky Wikidata reconciliation service currently, I managed to get a wide variety of errors with the reconciliation process then hanging and not being cancellable. We need to improve this.

Errors logged on OpenRefine console

07:42:26.185 [                   refine] POST /command/core/guess-types-of-column (42ms)
07:42:26.416 [..le.refine.model.Project] Loaded project 1943190398909 from disk in 0 sec(s) (230ms)
07:42:42.934 [                   refine] POST /command/core/reconcile (16518ms)
07:52:49.606 [..ic.HttpRequestRetryExec] ex-0000000067 recoverable I/O exception (org.apache.hc.core5.http.NoHttpResponseException) caught when sending request to http://localhost:8000;request will be automatically re-executed in 0 MILLISECONDS (exec count 2) (606671ms)
07:53:40.922 [..ic.HttpRequestRetryExec] ex-0000000084 recoverable I/O exception (org.apache.hc.core5.http.NoHttpResponseException) caught when sending request to http://localhost:8000;request will be automatically re-executed in 0 MILLISECONDS (exec count 2) (51315ms)
07:56:25.254 [..e.io.FileProjectManager] Saving some modified projects ... (164332ms)
07:56:25.686 [        project_utilities] Saved project '1943190398909' (432ms)
08:18:54.243 [                   refine] GET /command/core/get-csrf-token (1348556ms)
08:18:54.334 [                   refine] POST /command/core/cancel-processes (90ms)
08:18:54.782 [..le.refine.model.Project] Loaded project 1943190398909 from disk in 0 sec(s) (448ms)
08:18:54.860 [                   refine] GET /command/core/get-history (77ms)

I'm not sure where the NoHttpResponse exceptions came from, but they might have been involved in killing the reconciliation task.

Although the cancel-processes command seems to have mostly worked, the front end was still spinning with the "Canceling..." dialog up.

Errors populated in the recon cells included:

• No. of recon objects was less than no. of jobs
• HTTP error 403 : for URL /en/api [sic] - this should be fixed to return the full path with query
  = The reconciliation service returned an invalid response

I think the first two are variations due to how the calls are batched with the underlying error being a 403 in both cases.

The middle one might have been from this error that the reconciliation service got:

reconcile-1  |   File "/openrefine-wikibase/wdreconcile/engine.py", line 57, in _srsearch
reconcile-1  |     resp = await r.json()
reconcile-1  |   File "/usr/local/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 1109, in json
reconcile-1  |     headers=self.headers,
reconcile-1  | aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html; charset=utf-8', url=URL('https://www.wikidata.org/w/api.php?action=query&format=json&list=search&srnamespace=0&srlimit=50&srsearch=Susan+Mallery&srwhat=text')
where it attempted a JSON decode before checking the response status code.

To Reproduce

Steps to reproduce the behavior:

1. First, do ...
2. Then, ...
3. Finally, ...

Current Results

Expected Behavior

Screenshots

Versions

• Operating System:
• Browser Version:
• JRE or JDK Version:
• OpenRefine:

Datasets

Additional context