MeltanoLabs / MeltanoLabs/tap-github
Stream schema does not seem to be respected in produced records
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 20
- Forks
- 60
- Avg merge
- 20h 29m
- Merged PRs (30d)
- 6
Description
I might be misunderstanding how the schema definition works for a stream, but this bothers me.
With the following schema (from issue_comments in 71b07b7ba4cdfc13f7a2c651252d163206e5c56f):
schema = th.PropertiesList(
th.Property("id", th.IntegerType),
th.Property("node_id", th.StringType),
th.Property("repo", th.StringType),
th.Property("org", th.StringType),
th.Property("issue_url", th.IntegerType),
th.Property("updated_at", th.DateTimeType),
th.Property("created_at", th.DateTimeType),
th.Property("author_association", th.StringType),
th.Property("body", th.StringType),
th.Property(
"user",
th.ObjectType(
th.Property("login", th.StringType),
th.Property("id", th.IntegerType),
th.Property("node_id", th.StringType),
th.Property("avatar_url", th.StringType),
th.Property("gravatar_id", th.StringType),
th.Property("html_url", th.StringType),
th.Property("type", th.StringType),
th.Property("site_admin", th.BooleanType),
),
),
).to_dict()
I am seeing the following records:
{
"type": "RECORD",
"stream": "issue_comments",
"record": {
"issue_url": "https://api.github.com/repos/singer-io/tap-facebook/issues/157",
"id": 895733451,
"node_id": "IC_kwDOBRvyIM41Y87L",
"user": {
"login": "lscottallen004",
"id": 83940128,
"node_id": "MDQ6VXNlcjgzOTQwMTI4",
"avatar_url": "https://avatars.githubusercontent.com/u/83940128?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/lscottallen004",
"html_url": "https://github.com/lscottallen004",
"followers_url": "https://api.github.com/users/lscottallen004/followers",
"following_url": "https://api.github.com/users/lscottallen004/following{/other_user}",
"gists_url": "https://api.github.com/users/lscottallen004/gists{/gist_id}",
"starred_url": "https://api.github.com/users/lscottallen004/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/lscottallen004/subscriptions",
"organizations_url": "https://api.github.com/users/lscottallen004/orgs",
"repos_url": "https://api.github.com/users/lscottallen004/repos",
"events_url": "https://api.github.com/users/lscottallen004/events{/privacy}",
"received_events_url": "https://api.github.com/users/lscottallen004/received_events",
"type": "User",
"site_admin": false
},
"created_at": "2021-08-10T05:09:01Z",
"updated_at": "2021-08-10T05:09:01Z",
"author_association": "NONE",
"body": "> \n\n- > > > > > > ~~> >~~~~~~~~> @iterati _@iterati _@iterati _[]()[]()_@hz-lschick ____>~~ []",
"org": "singer-io",
"repo": "tap-facebook"
},
"time_extracted": "2021-09-10T23:07:19.736119Z"
}
A number of the nested *_url fields are present in the record, although they are excluded from the schema definition. It looks like a call to https://gitlab.com/meltano/sdk/-/blob/main/singer_sdk/helpers/_singer.py#L23 from pop_deselected_record_properties causes the field to be included because user is included, and somehow the details of the nested object do not appear in the selection mask.
I suspect this is a bug in the sdk, but I might have misunderstood how the code is supposed to work 🤔
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with pop_deselected_record_properties in singer_sdk/helpers/_singer.py and reproduce the issue using the issue_comments schema shown here. Check how the nested user selection mask is built; done when fields such as the nested *_url properties that are absent from the schema are omitted from produced records.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100