aiondemand / aiondemand/aiondemand

[ENH] Add get_all() convenience method for transparent auto-pagination across all asset endpoints

Ouverte
#135 2 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Python
Étoiles
38
Forks
89
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

`get_list()` returns at most one page of results (controlled by `offset` and `limit`). Users who want to retrieve an entire asset type from the catalogue must write their own pagination loop — a repetitive, error-prone pattern that the SDK should abstract away.

## Motivation

The typical workaround today looks like this:
```python
import aiod

all_results = []
offset = 0
limit = 100
while True:
page = aiod.datasets.get_list(offset=offset, limit=limit, data_format="json")
if not page:
break
all_results.extend(page)
offset += limit
```

This boilerplate recurs in any connector, export script, or research pipeline that needs the full catalogue. It belongs in the SDK, not in every user's code.

## Proposed API

Add a `get_all()` method alongside `get_list()` on every asset resource class (`datasets`, `publications`, `ml_models`, etc.):
```python
# Returns a full DataFrame across all pages
aiod.datasets.get_all()

# With options
aiod.publications.get_all(batch_size=50, data_format="json", show_progress=True)
```

Proposed signature:
```python
def get_all(
self,
batch_size: int = 100,
data_format: Literal["dataframe", "json"] = "dataframe",
show_progress: bool = False,
) -> pd.DataFrame | list[dict]:
...
```

- `batch_size` controls how many records are fetched per internal request
- `show_progress` optionally renders a progress bar (can reuse the tqdm approach from #107)
- All pages are concatenated into a single DataFrame (or list) before returning
- Iteration stops when the API returns an empty page

## Implementation Notes

- Lives alongside `get_list()` in the base resource class
- Should use the existing HTTP client (sync or async path)
- Should be resilient — deferring to the retry mechanism from #92 if available
- Reuses the shared session from #95 for efficiency
- Tests should mock multi-page responses to verify correct pagination and concatenation

## Why This Is Useful

- Connectors enumerating existing assets to detect duplicates need the full catalogue
- Researchers building datasets of AIoD metadata want all records without boilerplate
- Anyone writing an export or backup script benefits from a clean one-liner

## Related Issues / PRs

- #107 / #113 — progress bar (can integrate with `show_progress`)
- #92 — retry mechanism (useful for robustness over many pages)
- #95 — shared session reuse (efficiency for many sequential requests)

---

I'd love to pick this up if the maintainers are open to it. Happy to discuss the design before starting, adjust the API shape to fit the codebase conventions, or scope it down if a simpler version is preferred first.

cc @fkiraly would love to know your thoughts on this!

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.