dlt-hub / dlt-hub/dlt

REST API: add initial_cursor to JSONResponseCursorPaginator

Open
#4,428 0 comments 0 reactions 1 assignee Claimed by @burnash View on GitHub
enhancement needs decision
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 14h
Merged PRs (30d)
38

Description

### Feature description

`JSONResponseCursorPaginator` has no way to send a cursor value on the first request. `OffsetPaginator` has `initial_offset` and `PageNumberPaginator` has `initial_page`. There is no cursor equivalent.

### Are you a dlt user?

Yes, I run dlt in production.

### Use case

Cursor APIs fall into two groups that differ only in the first request.

Some return the first token in response to a request with no cursor. dlt already handles these.

Others only return a cursor if the request already sent one. You send `cursor=`, or `cursor=0`, or `cursorMark=*` in Solr and Elasticsearch, to start. Send nothing and the response comes back with `"cursor": null`.

With the second group `JSONResponseCursorPaginator` sends no cursor, gets `null` back, and `update_state` reads that as the end of the data:

```py
self._next_reference = values[0] if values and values[0] else None
```

The resource yields page one and the pipeline succeeds. There is no error and the row count is not zero, so nothing shows that the rest of the data is missing. We ran into this writing a connector for a vendor whose docs say "pass an empty cursor query param to retrieve the first cursor".

### Proposed solution

An `initial_cursor` argument, applied in `init_request` through the existing `update_request` path so both `cursor_param` and `cursor_body_path` keep working:

```py
def init_request(self, request: Request) -> None:
super().init_request(request)
if self.initial_cursor is None:
return
self._next_reference = self.initial_cursor
self.update_request(request)
```

A default of `None` keeps current behaviour. The key also needs adding to `JSONResponseCursorPaginatorConfig` to be usable from dict config, as in #3548.

### Related issues

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.