MeltanoLabs / MeltanoLabs/tap-github
Consider running tap output into a target in CI
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 20
- Forks
- 60
- Avg merge
- 20h 29m
- Merged PRs (30d)
- 6
Description
In #93 I decided to run the tap thru a simple target target-sqlite to see if the new streams I added worked.
By doing this, I instantly found mistyped fields in the Events stream.
Example error:
CRITICAL '20730626430' is not of type 'integer', 'null'
CRITICAL
CRITICAL Failed validating 'type' in schema['properties']['id']:
CRITICAL {'type': ['integer', 'null']}
CRITICAL
CRITICAL On instance['id']:
CRITICAL '20730626430'
I think this would be a great testing method to ensure our schemas are correctly typed.
Possibly a new step that runs:
poetry run tap-github --config c.json --catalog ca.json | poetry run target-sqlite -c sq.json and crashes if the exit code is non-zero.
Thoughts on implementing this?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the proposed pipeline using c.json, ca.json, and sq.json: poetry run tap-github --config c.json --catalog ca.json | poetry run target-sqlite -c sq.json. Check how CI currently runs commands, then make the step fail on a non-zero exit code so schema type errors such as the Events stream's invalid id are detected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sqlite
- Domain
- ci-cd, testing-qa
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100