POST /api/maintenance/migrate throws socketException

Once cortex is installed from binary, we run /api/maintenance/migrate to bootstrap cortex index and super admin

Request Type

Bug

Work Environment

Openshift 4.3

Problem Description

Steps to Reproduce

1. Once cortex image is installed,
2. We run python script using HTTPSConnection.request for POST /api/maintenance/migrate
   3 it throws below error

Possible Solutions

Remedy explained here : https://github.com/sksamuel/elastic4s/issues/1433

Complementary information

[info] c.s.e.h.ElasticClient$ - Creating HTTP client on https://tis-elastic-es-http:9200
[info] c.s.e.h.ElasticClient$ - Creating HTTP client on https://tis-elastic-es-http:9200
[info] c.s.e.h.ElasticClient$ - Creating HTTP client on https://tis-elastic-es-http:9200
[info] o.e.s.MigrationSrv - Create a new empty database
[info] o.e.s.MigrationSrv - Migrate database from version 0, add operations for version 2
[info] o.e.s.MigrationSrv - Migrate database from version 0, add operations for version 3
[info] o.e.s.MigrationSrv - Migrate database from version 0, add operations for version 4
[error] o.e.s.MigrationSrv - Migration fail
com.sksamuel.elastic4s.http.JavaClientExceptionWrapper: java.net.SocketTimeoutException
at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient$$anon$1.onFailure(ElasticsearchJavaRestClient.scala:63)
at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:850)
at org.elasticsearch.client.RestClient$1.retryIfPossible(RestClient.java:588)
at org.elasticsearch.client.RestClient$1.failed(RestClient.java:567)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:134)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:419)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:375)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
Caused by: java.net.SocketTimeoutException: null
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:375)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:263)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:492)
at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:213)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)

[info] o.t.c.s.ErrorHandler - POST /api/maintenance/migrate returned 500
com.sksamuel.elastic4s.http.JavaClientExceptionWrapper: java.net.SocketTimeoutException
at com.sksamuel.elastic4s.http.ElasticsearchJavaRestClient$$anon$1.onFailure(ElasticsearchJavaRestClient.scala:63)
at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:850)
at org.elasticsearch.client.RestClient$1.retryIfPossible(RestClient.java:588)
at org.elasticsearch.client.RestClient$1.failed(RestClient.java:567)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:134)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:419)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:375)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)

Start at the POST /api/maintenance/migrate entry point and follow the MigrationSrv operations that create and migrate the database. Review the ElasticClient failure shown in the report and compare it with the linked elastic4s issue, using the Elasticsearch 6.8.8 environment described. Done means the migration completes without the reported SocketTimeoutException.