dlt-hub / dlt-hub/dlt

feat(rest_api): support `.add_limit()` for derived endpoints

Open
#3,099 3 comments 0 reactions 0 assignees View on GitHub
bug enhancement
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 14h
Merged PRs (30d)
38

Description

When using a REST API source that has endpoints dependencies such as `repos/dlt-hub/dlt/actions/workflows/{resources.workflows.id}/runs`, you can't use `.add_limit()`

```python
resources = [
{
"name": "workflows",
"endpoint": {
"path": f"repos/{owner}/{repo}/actions/workflows",
"data_selector": "workflows",
},
},
{
"name": "runs",
"endpoint": {
"path": f"repos/{owner}/{repo}/actions/workflows/{{resources.workflows.id}}/runs",
"data_selector": "workflow_runs"
}
},
]
```

You will get
```
...|dlt|resource.py|add_limit:404|Setting add_limit to a transformer runs has no effect. Set the limit on the top level resource.
```

>❗ it's possible that there is no issue / bug and the warning is a false positive. For example, `.add_limit()` is properly set on the parent `workflows`, but it is also set on the child `workflows_runs`, which triggers the warning

## Problem
The user-facing function `rest_api_resources()` returns resources already piped. This is done internally by `create_resources()` which uses the `data_from` kwarg of `@dlt.resource()` to instantiate transformers.

Then, using `.add_limit()` on the returned `dlt.resource` doesn't successfully update the relationships the other resources's pipe

## Todo
- figure out how `.add_limit()` mutates resources in place and how it's propagated to transformer relationships (this should be accessible in a `DltResource.pipe` attribute)
- Update and improve the warning. The message `Setting add_limit to a transformer runs` is not useful given the user didn't set these things. The warning message should better describe the solution

## Related
- similar issue to built-in file readers sources: https://github.com/dlt-hub/dlt/issues/2858

Contributor guide

Open the contributing guide

Research direction

Start by tracing rest_api_resources() through create_resources(), focusing on the data_from argument, then inspect how add_limit() mutates resources and how relationships are exposed through DltResource.pipe. Done means limits propagate correctly to derived endpoints and the warning explains the appropriate solution; compare the behavior with the related built-in file readers issue #2858.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.