MeltanoLabs / MeltanoLabs/tap-gitlab

Tap downloads all global gitlab users

Open
#91 4 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
12
Forks
29
Avg merge
4d 1h
Merged PRs (30d)
1

Description

Hi there!

when attempting to get a single project, tap-gitlab attempts to download ALL users of gitlab.

In my case, I'm forwarding this to the target-gsheet and fill a google sheet with the project info. This is a free gitlab account, no pro license, no custom URL.
I am running the following command

tap-gitlab -c config_gitlab.json | target-gsheet -c config_gsheet.json

My config_gitlab.json looks like this

{
    "api_url": "https://gitlab.com",
    "private_token": "abcdefgandsoon",
    "groups": "mygroup",
    "projects": "mygroup/myproject",
    "start_date": "2018-01-01T00:00:00Z",
    "ultimate_license": false,
    "fetch_merge_request_commits": false,
    "fetch_pipelines_extended": false,
    "fetch_group_variables": false,
    "fetch_project_variables": false
}

Upon running the command I get this output

INFO Starting sync
INFO Skipping stream: merge_request_commits
INFO Skipping stream: epics
INFO Skipping stream: epic_issues
INFO Skipping stream: pipelines_extended
INFO GET https://gitlab.com/api/v4/users
INFO GET https://gitlab.com/api/v4/users

...after which a 'site_users' panel appears in the google sheet which is filled with endless rows of users.

Am I doing something wrong so that tap-gitlab doesn't filter users by group?

Or is there an option to skip users alltogether (I don't actually need them)

Best
Benjamin

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the command with the supplied config and observe the site_users stream and repeated GET https://gitlab.com/api/v4/users requests. Trace where that stream is enabled or scoped, then verify that a single project does not download all GitLab users or that users can be skipped when they are not needed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.