REST API: OffsetPaginator raises on null total even when stop_after_empty_page is set
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 605
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
### dlt version
1.30.0
### Describe the problem
`RangePaginator.update_state` raises when `total_path` resolves to `None`, even when `stop_after_empty_page` is `True` and would have ended the walk correctly.
`total_path` defaults to `"total"` on `OffsetPaginator` and `PageNumberPaginator`, and `stop_after_empty_page` defaults to `True`. So for an API whose response has no `total`, or has `"total": null`, the paginator raises on the first non-empty page unless you know to pass `total_path=None`.
`"total": null` is the API saying it does not know the total, which is different from a response that left the field out by mistake. We hit this on a vendor that returns a number for `total` on some endpoints and `null` on others, with the same envelope on both.
The part that costs time is that `stop_after_empty_page=True` is both the default and a complete stop condition, but it does not prevent the raise. The two settings look like they overlap and they do not.
### Expected behavior
If `total_path` gives no usable value but another stop condition is set (`stop_after_empty_page`, `maximum_value` or `has_more_path`), use that and log a warning instead of raising.
If raising is preferred, the error should mention `total_path=None` as the fix. That is not obvious from the current message.
### Steps to reproduce
```py
from dlt.sources.helpers.rest_client.paginators import OffsetPaginator
class Response:
@staticmethod
def json():
return {"data": [{"id": 1}], "total": None}
paginator = OffsetPaginator(limit=100, stop_after_empty_page=True)
paginator.update_state(Response(), data=[{"id": 1}])
```
```
ValueError: Total `items` not found in the response in `OffsetPaginator`.
Expected a response with a `total` key, got `{'data': [{'id': 1}], 'total': None}`.
```
Passing `total_path=None` works as expected.
### Operating system
macOS
### Runtime environment
Local
### Python version
3.12
### dlt data source
REST API
### dlt destination
_No response_
### Other deployment details
_No response_
### Additional information
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.