MeltanoLabs / MeltanoLabs/tap-postgres

bug: `jsonschema.exceptions.ValidationError` when loading `information_schema`

Open
#437 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug help wanted
Dominant language
Python
Stars
26
Forks
42
PR merge metrics
No merged PRs in 30d

Description

Overview

Trying to debug this issue from Slack and getting a jsonschema.exceptions.ValidationError when loading information_schema views with target-jsonl:

2024-06-06 23:40:34,704 | INFO     | tap-postgres.information_schema-columns | Beginning full_table sync of 'information_schema-columns'...
2024-06-06 23:40:34,704 | INFO     | tap-postgres.information_schema-columns | Tap has custom mapper. Using 1 provided map(s).
Traceback (most recent call last):
  File "/home/reuben/Documents/taps/tap-postgres/.meltano/loaders/target-jsonl/venv/bin/target-jsonl", line 8, in <module>
    sys.exit(main())
  File "/home/reuben/Documents/taps/tap-postgres/.meltano/loaders/target-jsonl/venv/lib/python3.8/site-packages/target_jsonl.py", line 92, in main
    state = persist_messages(
  File "/home/reuben/Documents/taps/tap-postgres/.meltano/loaders/target-jsonl/venv/lib/python3.8/site-packages/target_jsonl.py", line 54, in persist_messages
    validators[o['stream']].validate((o['record']))
  File "/home/reuben/Documents/taps/tap-postgres/.meltano/loaders/target-jsonl/venv/lib/python3.8/site-packages/jsonschema/validators.py", line 130, in validate
    raise error
jsonschema.exceptions.ValidationError: 1 is not of type 'string', 'null'

Failed validating 'type' in schema['properties']['ordinal_position']:
    {'type': ['string', 'null']}

On instance['ordinal_position']:
    1
2024-06-06 23:40:35,979 | INFO     | singer_sdk.metrics   | METRIC: {"type": "timer", "metric": "sync_duration", "value": 1.2741827964782715, "tags": {"stream": "information_schema-columns", "context": {}, "status": "failed"}}
2024-06-06 23:40:35,979 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "record_count", "value": 49, "tags": {"stream": "information_schema-columns", "context": {}}}
2024-06-06 23:40:35,979 | ERROR    | tap-postgres.information_schema-columns | An unhandled error occurred while syncing 'information_schema-columns'
Traceback (most recent call last):
  File "/home/reuben/Documents/taps/tap-postgres/.meltano/extractors/tap-postgres/venv/lib/python3.8/site-packages/singer_sdk/streams/core.py", line 1190, in sync
    for _ in self._sync_records(context=context):
  File "/home/reuben/Documents/taps/tap-postgres/.meltano/extractors/tap-postgres/venv/lib/python3.8/site-packages/singer_sdk/streams/core.py", line 1113, in _sync_records
    self._write_record_message(record)
  File "/home/reuben/Documents/taps/tap-postgres/.meltano/extractors/tap-postgres/venv/lib/python3.8/site-packages/singer_sdk/streams/core.py", line 856, in _write_record_message
    self._tap.write_message(record_message)
  File "/home/reuben/Documents/taps/tap-postgres/.meltano/extractors/tap-postgres/venv/lib/python3.8/site-packages/singer_sdk/io_base.py", line 164, in write_message
    singer_write_message(message)
  File "/home/reuben/Documents/taps/tap-postgres/.meltano/extractors/tap-postgres/venv/lib/python3.8/site-packages/singer_sdk/_singerlib/messages.py", line 244, in write_message
    sys.stdout.flush()
BrokenPipeError: [Errno 32] Broken pipe

Reproduce
docker run --rm -e POSTGRES_HOST_AUTH_METHOD=trust -p 5432:5432 postgres
git clone git@github.com:MeltanoLabs/tap-postgres.git
cd tap-postgres
meltano install
meltano invoke tap-postgres | meltano invoke target-jsonl
Workaround

Deselect the information_schema streams (related to #54):

meltano select tap-postgres --exclude 'information_schema-*'
meltano select tap-postgres --all

Or just select the public schema:

meltano select tap-postgres 'public-*'

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the documented PostgreSQL Docker command and the tap-postgres-to-target-jsonl pipeline. Trace the information_schema-columns stream, including its schema and custom mapper, to determine why ordinal_position is emitted as an integer; done means the stream loads successfully without the jsonschema ValidationError.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.