Question regarding handling of HTTP 429 with reactive stream

Hi,

we are using spring-data-elasticsearch with version 4.3.0. Our backend is opensearch v1. Whenever we do too many requests opensearch sends HTTP 429 to throttle the client and to tell us that it was too much. Our current approach to that is to retry it, so we have something like that:

repository.mySearchFunctionWhichReturnsFluxOfEntity() .retryExponentialBackoff(20, Duration.ofSeconds(1), Duration.ofSeconds(20), false) { log.debug("I retried") } .doOnNext { // do something with the entity, in our case: add them to a report class to be written to the file system }

Now ... my expectation was that the retry will only retry the failed scroll call, but it doesn't, it restarts the whole flux.
So instead of having 5M rows in our report we now have 25M rows, because it restarted 18 times and so the creation of the report took 7h instead of 45min.

I really don't want to use elasticsearch client lib directly to have full control over my SearchScroll and cover all of the retriable stuff directly before adding them to the Flux, so I would like to know how this can be solved with the tools that spring-data-elasticsearch provide, or something that configures the underlying code.
I thought that this must occur very often and so I think there must be a way to not restart the stream or at least restart it where it broke.

The entry point is repository.mySearchFunctionWhichReturnsFluxOfEntity(), with retryExponentialBackoff and doOnNext in the report flow. Start by tracing how the reactive scroll request is exposed and how retry resubscribes after HTTP 429. Done should be defined as retry behavior that avoids duplicating already emitted rows.