MeltanoLabs / MeltanoLabs/tap-gitlab
tap-gitlab - sync_pipelines_extended is failing due to difference in schema
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 12
- Forks
- 29
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 1
Description
Error:
Traceback (most recent call last):
File "/Users/amalkumar/venv/bin/tap-gitlab", line 11, in
load_entry_point('tap-gitlab==0.9.15', 'console_scripts', 'tap-gitlab')()
File "/Users/amalkumar/venv/lib/python3.7/site-packages/tap_gitlab/init.py", line 959, in main
raise exc
File "/Users/amalkumar/venv/lib/python3.7/site-packages/tap_gitlab/init.py", line 956, in main
main_impl()
File "/Users/amalkumar/venv/lib/python3.7/site-packages/tap_gitlab/init.py", line 951, in main_impl
do_sync()
File "/Users/amalkumar/venv/lib/python3.7/site-packages/tap_gitlab/init.py", line 904, in do_sync
sync_group(gid, pids)
File "/Users/amalkumar/venv/lib/python3.7/site-packages/tap_gitlab/init.py", line 679, in sync_group
sync_project(pid)
File "/Users/amalkumar/venv/lib/python3.7/site-packages/tap_gitlab/init.py", line 834, in sync_project
sync_pipelines(data)
File "/Users/amalkumar/venv/lib/python3.7/site-packages/tap_gitlab/init.py", line 723, in sync_pipelines
sync_pipelines_extended(project, transformed_row)
File "/Users/amalkumar/venv/lib/python3.7/site-packages/tap_gitlab/init.py", line 744, in sync_pipelines_extended
transformed_row = transformer.transform(row, RESOURCES[entity]["schema"], mdata)
File "/Users/amalkumar/venv/lib/python3.7/site-packages/singer/transform.py", line 152, in transform
raise SchemaMismatch(self.errors)
singer.transform.SchemaMismatch: Errors during transform
user: data does not match {'type': 'object', 'properties': {'name': {'type': 'string'}, 'username': {'type': 'string'}, 'id': {'type': 'integer'}, 'state': {'type': 'string'}}}
committed_at: data does not match {'type': 'string', 'format': 'date-time'}
coverage: data does not match {'type': 'number'}
: data does not match {'type': 'object', 'properties': {'project_id': {'type': ['integer', 'null']}, 'id': {'type': ['integer', 'null']}, 'status': {'type': ['string', 'null']}, 'ref': {'type': ['string', 'null']}, 'sha': {'type': ['string', 'null']}, 'before_sha': {'type': ['string', 'null']}, 'tag': {'type': ['boolean', 'null']}, 'yaml_errors': {'type': ['string', 'null']}, 'user': {'type': 'object', 'properties': {'name': {'type': 'string'}, 'username': {'type': 'string'}, 'id': {'type': 'integer'}, 'state': {'type': 'string'}}}, 'created_at': {'anyOf': [{'type': 'string', 'format': 'date-time'}, {'type': 'null'}]}, 'updated_at': {'anyOf': [{'type': 'string', 'format': 'date-time'}, {'type': 'null'}]}, 'started_at': {'anyOf': [{'type': 'string', 'format': 'date-time'}, {'type': 'null'}]}, 'finished_at': {'anyOf': [{'type': 'string', 'format': 'date-time'}, {'type': 'null'}]}, 'committed_at': {'anyOf': [{'type': 'string', 'format': 'date-time'}, {'type': 'null'}]}, 'duration': {'anyOf': [{'type': 'integer'}, {'type': 'null'}]}, 'coverage': {'anyOf': [{'type': 'number'}, {'type': 'null'}]}, 'web_url': {'type': ['string', 'null']}}}
Errors during transform: [user: data does not match {'type': 'object', 'properties': {'name': {'type': 'string'}, 'username': {'type': 'string'}, 'id': {'type': 'integer'}, 'state': {'type': 'string'}}}, committed_at: data does not match {'type': 'string', 'format': 'date-time'}, coverage: data does not match {'type': 'number'}, : data does not match {'type': 'object', 'properties': {'project_id': {'type': ['integer', 'null']}, 'id': {'type': ['integer', 'null']}, 'status': {'type': ['string', 'null']}, 'ref': {'type': ['string', 'null']}, 'sha': {'type': ['string', 'null']}, 'before_sha': {'type': ['string', 'null']}, 'tag': {'type': ['boolean', 'null']}, 'yaml_errors': {'type': ['string', 'null']}, 'user': {'type': 'object', 'properties': {'name': {'type': 'string'}, 'username': {'type': 'string'}, 'id': {'type': 'integer'}, 'state': {'type': 'string'}}}, 'created_at': {'anyOf': [{'type': 'string', 'format': 'date-time'}, {'type': 'null'}]}, 'updated_at': {'anyOf': [{'type': 'string', 'format': 'date-time'}, {'type': 'null'}]}, 'started_at': {'anyOf': [{'type': 'string', 'format': 'date-time'}, {'type': 'null'}]}, 'finished_at': {'anyOf': [{'type': 'string', 'format': 'date-time'}, {'type': 'null'}]}, 'committed_at': {'anyOf': [{'type': 'string', 'format': 'date-time'}, {'type': 'null'}]}, 'duration': {'anyOf': [{'type': 'integer'}, {'type': 'null'}]}, 'coverage': {'anyOf': [{'type': 'number'}, {'type': 'null'}]}, 'web_url': {'type': ['string', 'null']}}}]
Steps to reproduce:
API version: 0.9.15 & 0.10.0
Python: 3.7.3
Config:
{
"api_url": "https://",
"private_token": "",
"groups": "",
"projects": "",
"start_date": "",
"ultimate_license": true,
"fetch_merge_request_commits": true,
"fetch_pipelines_extended": true
}
Command:
tap-gitlab --config tap-gitlab-config
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the tap-gitlab entry point and the sync_pipelines_extended path shown in tap_gitlab/init.py, then inspect the pipeline schema and transformation inputs involved in the reported Singer SchemaMismatch. Reproduce with fetch_pipelines_extended enabled and verify that the pipeline records match the declared schema without errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100