feat(rest_api): support `.add_limit()` for derived endpoints
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 600
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
When using a REST API source that has endpoints dependencies such as `repos/dlt-hub/dlt/actions/workflows/{resources.workflows.id}/runs`, you can't use `.add_limit()`
```python
resources = [
{
"name": "workflows",
"endpoint": {
"path": f"repos/{owner}/{repo}/actions/workflows",
"data_selector": "workflows",
},
},
{
"name": "runs",
"endpoint": {
"path": f"repos/{owner}/{repo}/actions/workflows/{{resources.workflows.id}}/runs",
"data_selector": "workflow_runs"
}
},
]
```
You will get
```
...|dlt|resource.py|add_limit:404|Setting add_limit to a transformer runs has no effect. Set the limit on the top level resource.
```
>❗ it's possible that there is no issue / bug and the warning is a false positive. For example, `.add_limit()` is properly set on the parent `workflows`, but it is also set on the child `workflows_runs`, which triggers the warning
## Problem
The user-facing function `rest_api_resources()` returns resources already piped. This is done internally by `create_resources()` which uses the `data_from` kwarg of `@dlt.resource()` to instantiate transformers.
Then, using `.add_limit()` on the returned `dlt.resource` doesn't successfully update the relationships the other resources's pipe
## Todo
- figure out how `.add_limit()` mutates resources in place and how it's propagated to transformer relationships (this should be accessible in a `DltResource.pipe` attribute)
- Update and improve the warning. The message `Setting add_limit to a transformer runs` is not useful given the user didn't set these things. The warning message should better describe the solution
## Related
- similar issue to built-in file readers sources: https://github.com/dlt-hub/dlt/issues/2858
Contributor guide
Research direction
Start by tracing rest_api_resources() through create_resources(), focusing on the data_from argument, then inspect how add_limit() mutates resources and how relationships are exposed through DltResource.pipe. Done means limits propagate correctly to derived endpoints and the warning explains the appropriate solution; compare the behavior with the related built-in file readers issue #2858.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100