singer-io / singer-io/tap-gitlab
When use api/v4, will only get the last 20 records
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 16
- Forks
- 18
- PR merge metrics
- No merged PRs in 30d
Description
After check the code, I found the header of api/v4 has not contains "X-Total-Pages", so we cannot get the total page. The "X-Next-Page" is a good solution, if the records are read to end, this attribute will empty.
params = {'page': 1}
resp = request(url, params)
for row in resp.json():
yield row
next_page_s = resp.headers.get('X-Next-Page', 1)
while len(next_page_s) != 0:
next_page_i = int(next_page_s)
params['page'] = next_page_i
resp = request(url, params)
next_page_s = resp.headers.get('X-Next-Page', 1)
for row in resp.json():
yield row
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the Python code that requests GitLab api/v4 records and handles pagination. Reproduce a response containing X-Next-Page, then verify that iteration continues until the header is empty and all records are yielded; add or update coverage if the repository already has tests for this path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gitlab, python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100