Null Only Schema Objects

Motivation

While trying to run tap-github against https://github.com/datamill-co/target-postgres I got the below errors.

The issue from what I can tell is that under the issues stream, the values of milestone and closed_at have the ill fated "type": ["null"] conundrum. This is something that is fairly common when using the https://github.com/singer-io/singer-tools schema detection.

Errors

Errors during transform: [
  milestone: {'url': 'https://api.github.com/repos/datamill-co/target-postgres/milestones/1', 'html_url': 'https://github.com/datamill-co/target-postgres/milestone/1', 'labels_url': 'https://api.github.com/repos/datamill-co/target-postgres/milestones/1/labels', 'id': 3881167, 'node_id': 'MDk6TWlsZXN0b25lMzg4MTE2Nw==', 'number': 1, 'title': 'Stabilize V0', 'description': None, 'creator': {'login': 'AlexanderMann', 'id': 3885029, 'node_id': 'MDQ6VXNlcjM4ODUwMjk=', 'avatar_url': 'https://avatars0.githubusercontent.com/u/3885029?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/AlexanderMann', 'html_url': 'https://github.com/AlexanderMann', 'followers_url': 'https://api.github.com/users/AlexanderMann/followers', 'following_url': 'https://api.github.com/users/AlexanderMann/following{/other_user}', 'gists_url': 'https://api.github.com/users/AlexanderMann/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/AlexanderMann/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/AlexanderMann/subscriptions', 'organizations_url': 'https://api.github.com/users/AlexanderMann/orgs', 'repos_url': 'https://api.github.com/users/AlexanderMann/repos', 'events_url': 'https://api.github.com/users/AlexanderMann/events{/privacy}', 'received_events_url': 'https://api.github.com/users/AlexanderMann/received_events', 'type': 'User', 'site_admin': False}, 'open_issues': 1, 'closed_issues': 0, 'state': 'open', 'created_at': '2018-12-07T22:33:34Z', 'updated_at': '2018-12-07T22:42:13Z', 'due_on': None, 'closed_at': None}

  does not match {'type': ['null']}, :

  {'url': 'https://api.github.com/repos/datamill-co/target-postgres/issues/61', 'repository_url': 'https://api.github.com/repos/datamill-co/target-postgres', 'labels_url': 'https://api.github.com/repos/datamill-co/target-postgres/issues/61/labels{/name}', 'comments_url': 'https://api.github.com/repos/datamill-co/target-postgres/issues/61/comments', 'events_url': 'https://api.github.com/repos/datamill-co/target-postgres/issues/61/events', 'html_url': 'https://github.com/datamill-co/target-postgres/issues/61', 'id': 388839887, 'node_id': 'MDU6SXNzdWUzODg4Mzk4ODc=', 'number': 61, 'title': 'Stabilize V0 of `target-postgres`', 'user': {'login': 'AlexanderMann', 'id': 3885029, 'node_id': 'MDQ6VXNlcjM4ODUwMjk=', 'avatar_url': 'https://avatars0.githubusercontent.com/u/3885029?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/AlexanderMann', 'html_url': 'https://github.com/AlexanderMann', 'followers_url': 'https://api.github.com/users/AlexanderMann/followers', 'following_url': 'https://api.github.com/users/AlexanderMann/following{/other_user}', 'gists_url': 'https://api.github.com/users/AlexanderMann/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/AlexanderMann/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/AlexanderMann/subscriptions', 'organizations_url': 'https://api.github.com/users/AlexanderMann/orgs', 'repos_url': 'https://api.github.com/users/AlexanderMann/repos', 'events_url': 'https://api.github.com/users/AlexanderMann/events{/privacy}', 'received_events_url': 'https://api.github.com/users/AlexanderMann/received_events', 'type': 'User', 'site_admin': False}, 'labels': [{'id': 861129279, 'node_id': 'MDU6TGFiZWw4NjExMjkyNzk=', 'url': 'https://api.github.com/repos/datamill-co/target-postgres/labels/enhancement', 'name': 'enhancement', 'color': 'a2eeef', 'default': True}], 'state': 'open', 'locked': False, 'assignee': None, 'assignees': [], 'milestone': {'url': 'https://api.github.com/repos/datamill-co/target-postgres/milestones/1', 'html_url': 'https://github.com/datamill-co/target-postgres/milestone/1', 'labels_url': 'https://api.github.com/repos/datamill-co/target-postgres/milestones/1/labels', 'id': 3881167, 'node_id': 'MDk6TWlsZXN0b25lMzg4MTE2Nw==', 'number': 1, 'title': 'Stabilize V0', 'description': None, 'creator': {'login': 'AlexanderMann', 'id': 3885029, 'node_id': 'MDQ6VXNlcjM4ODUwMjk=', 'avatar_url': 'https://avatars0.githubusercontent.com/u/3885029?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/AlexanderMann', 'html_url': 'https://github.com/AlexanderMann', 'followers_url': 'https://api.github.com/users/AlexanderMann/followers', 'following_url': 'https://api.github.com/users/AlexanderMann/following{/other_user}', 'gists_url': 'https://api.github.com/users/AlexanderMann/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/AlexanderMann/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/AlexanderMann/subscriptions', 'organizations_url': 'https://api.github.com/users/AlexanderMann/orgs', 'repos_url': 'https://api.github.com/users/AlexanderMann/repos', 'events_url': 'https://api.github.com/users/AlexanderMann/events{/privacy}', 'received_events_url': 'https://api.github.com/users/AlexanderMann/received_events', 'type': 'User', 'site_admin': False}, 'open_issues': 1, 'closed_issues': 0, 'state': 'open', 'created_at': '2018-12-07T22:33:34Z', 'updated_at': '2018-12-07T22:42:13Z', 'due_on': None, 'closed_at': None}, 'comments': 0, 'created_at': '2018-12-07T22:42:13Z', 'updated_at': '2018-12-07T22:42:13Z', 'closed_at': None, 'author_association': 'COLLABORATOR', 'body': '# Motivation\r\n\r\n`Target-Postgres` has been changing dramatically for some time. While _many_ of these changes are bug fixes, or straight features, many are modifying the underlying metadata/schema and are not purely "breaking" from a "can I rebase my branch onto this" standpoint.\r\n\r\nAs such, I\'m proposing a new Milestone/Issue to gather up what bugs and housekeeping we want to get in place before going under a semvar versioning guide for ***persisted metadata/schema/data ONLY***. ie, we still will not have to play nice with those who wish to write _code_ on top of our repo, but we _will_ play nice with future versions of our code and we _will_ support the data which is persisted by v0.\r\n\r\n## Notes\r\n\r\nPlease understand that while I am in full support of making life simpler for other devs to work on top of our work, I\'m much more concerned about Datamill or someone trying to use this and then getting a tonne of data moved over...which then has to be blown away because we don\'t know how to migrate etc.\r\n\r\n## Outstanding Work\r\n\r\nOn the top of my mind, the work I think that needs to be done:\r\n\r\n- [ ] in flight feature work:\r\n  - [ ] https://github.com/datamill-co/target-postgres/issues/52\r\n- [ ] refactor table mappings and column mappings to use similar/same structure\r\n  - [ ] #44 \r\n- [ ] nested support as noted in:\r\n  - [ ] #42 \r\n  - [ ] #58 \r\n- [ ] Better testing\r\n  - [ ] Full integration test against Github/BigCommerce/whatever in CI (could be low hanging fruit for bugs etc.)\r\n\r\n## Suggested Musical Pairing\r\n\r\nhttps://soundcloud.com/hotchip/huarache-lights', '_sdc_repository': 'datamill-co/target-postgres'}

  does not match {'selected': True, 'properties': {'state': {'type': ['string', 'null']}, 'url': {'type': ['string', 'null']}, 'labels': {'type': ['array', 'null'], 'items': {'type': 'object', 'properties': {'id': {'type': ['integer', 'null']}, 'node_id': {'type': ['string', 'null']}, 'url': {'type': ['string', 'null']}, 'name': {'type': ['string', 'null']}, 'description': {'type': ['null', 'string']}, 'color': {'type': ['string', 'null']}, 'default': {'type': ['boolean', 'null']}}}}, 'repository_url': {'type': ['string', 'null']}, 'number': {'type': ['integer', 'null']}, 'closed_at': {'type': ['null']}, 'labels_url': {'type': ['string', 'null']}, 'title': {'type': ['string', 'null']}, 'assignee': {'type': ['object', 'null'], 'properties': {}}, 'updated_at': {'type': ['string', 'null']}, 'html_url': {'type': ['string', 'null']}, 'author_association': {'type': ['string', 'null']}, 'locked': {'type': ['boolean', 'null']}, 'events_url': {'type': ['string', 'null']}, 'pull_request': {'properties': {'diff_url': {'type': ['string', 'null']}, 'html_url': {'type': ['string', 'null']}, 'patch_url': {'type': ['string', 'null']}, 'url': {'type': ['string', 'null']}}, 'type': ['object', 'null']}, 'node_id': {'type': ['string', 'null']}, 'body': {'type': ['string', 'null']}, 'comments': {'type': ['integer', 'null']}, 'created_at': {'type': ['string', 'null']}, '_sdc_repository': {'type': ['string']}, 'user': {'properties': {'repos_url': {'type': ['string', 'null']}, 'starred_url': {'type': ['string', 'null']}, 'url': {'type': ['string', 'null']}, 'received_events_url': {'type': ['string', 'null']}, 'site_admin': {'type': ['boolean', 'null']}, 'gravatar_id': {'type': ['string', 'null']}, 'following_url': {'type': ['string', 'null']}, 'avatar_url': {'type': ['string', 'null']}, 'events_url': {'type': ['string', 'null']}, 'id': {'type': ['integer', 'null']}, 'login': {'type': ['string', 'null']}, 'organizations_url': {'type': ['string', 'null']}, 'html_url': {'type': ['string', 'null']}, 'type': {'type': ['string', 'null']}, 'subscriptions_url': {'type': ['string', 'null']}, 'node_id': {'type': ['string', 'null']}, 'followers_url': {'type': ['string', 'null']}, 'gists_url': {'type': ['string', 'null']}}, 'type': ['object', 'null']}, 'id': {'type': ['integer', 'null']}, 'comments_url': {'type': ['string', 'null']}, 'milestone': {'type': ['null']}}, 'type': ['object', 'null']}]

Suggested Musical Pairing

https://www.youtube.com/watch?v=vGhMdhdDIlM

Start with tap_github/issues.json, especially the schema entries for milestone and closed_at, and reproduce the transform against the target-postgres example described in the issue. The work is done when populated milestone and closed_at values are accepted while null values remain valid.