MeltanoLabs / MeltanoLabs/tap-github
Invalid SCHEMA messages are produced for deselected streams
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 20
- Forks
- 60
- Avg merge
- 20h 29m
- Merged PRs (30d)
- 6
Description
In #193, a set of `traffic_*` streams were added to the tap, with a customised `metadata` property, which deselects them if no catalog was passed as input to the tap.
Unfortunately, when running the tap with `poetry run tap-github --config /tmp/tmpmt8fq0pn/tmp7896kkwh.json --test=schema`
with this config (which does not seem to matter much, the main thing being the `test=schema` cli option):
```json
{"metrics_log_level": "error", "auth_token": "", "additional_auth_tokens": [], "rate_limit_buffer": 1000, "start_date": "2021-05-24 13:44:42.693145", "skip_parent_streams": true, "repositories": []}
```
the tap issues invalid `SCHEMA` messages like:
```json
{
"type": "SCHEMA",
"stream": "traffic_pageviews",
"schema": {"properties": {}, "type": "object"},
"key_properties": ["repo", "org", "timestamp"]
}
```
Specifically, `properties` is empty, so downstream targets cannot lookup the `key_properties`.
The line that causes the problem is here https://github.com/MeltanoLabs/tap-github/pull/193/files#diff-06dc9c6115cbc069ce355913de0c101fedf6956d6f6b4873c5112434596934d3R2260
I have not dug into the details yet, but it looks like the schema production does not correctly take the selection metadata into account.
Pinging @edgarrmondragon as you suggested that code, and you might have a fix for it :)
I also think the sdk should not allow a tap to produce invalid records like this. Is there a way to test against it without causing too much overhead? Obviously, we could validate each record before sending it out, but that might be a bit heavy ;)
Interestingly there's a test for this `_test_replication_keys_in_schema` but it does not validate against the schema messages that are sent.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the schema-production line identified in the #193 diff and reproduce the issue using the documented poetry run tap-github command and config. Read _test_replication_keys_in_schema, then trace how selection metadata affects the traffic_* streams. Done means schema messages have properties that support their key_properties, with coverage for the emitted schema messages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github, python
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100