dlt-hub / dlt-hub/dlt

REST API: OffsetPaginator raises on null total even when stop_after_empty_page is set

Open
#4,429 4 comments 0 reactions 1 assignee Claimed by @burnash View on GitHub
Dominant language
Python
Stars
5.9k
Forks
605
Avg merge
1d 14h
Merged PRs (30d)
38

Description

### dlt version

1.30.0

### Describe the problem

`RangePaginator.update_state` raises when `total_path` resolves to `None`, even when `stop_after_empty_page` is `True` and would have ended the walk correctly.

`total_path` defaults to `"total"` on `OffsetPaginator` and `PageNumberPaginator`, and `stop_after_empty_page` defaults to `True`. So for an API whose response has no `total`, or has `"total": null`, the paginator raises on the first non-empty page unless you know to pass `total_path=None`.

`"total": null` is the API saying it does not know the total, which is different from a response that left the field out by mistake. We hit this on a vendor that returns a number for `total` on some endpoints and `null` on others, with the same envelope on both.

The part that costs time is that `stop_after_empty_page=True` is both the default and a complete stop condition, but it does not prevent the raise. The two settings look like they overlap and they do not.

### Expected behavior

If `total_path` gives no usable value but another stop condition is set (`stop_after_empty_page`, `maximum_value` or `has_more_path`), use that and log a warning instead of raising.

If raising is preferred, the error should mention `total_path=None` as the fix. That is not obvious from the current message.

### Steps to reproduce

```py
from dlt.sources.helpers.rest_client.paginators import OffsetPaginator

class Response:
@staticmethod
def json():
return {"data": [{"id": 1}], "total": None}

paginator = OffsetPaginator(limit=100, stop_after_empty_page=True)
paginator.update_state(Response(), data=[{"id": 1}])
```

```
ValueError: Total `items` not found in the response in `OffsetPaginator`.
Expected a response with a `total` key, got `{'data': [{'id': 1}], 'total': None}`.
```

Passing `total_path=None` works as expected.

### Operating system

macOS

### Runtime environment

Local

### Python version

3.12

### dlt data source

REST API

### dlt destination

_No response_

### Other deployment details

_No response_

### Additional information

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.