MeltanoLabs / MeltanoLabs/tap-postgres

Unexpected `ValueError: Could not determine the key type` while testing the tap

Open
#241 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
26
Forks
42
PR merge metrics
No merged PRs in 30d

Description

Just getting started with Meltano, and testing this Postgres tap.

So far, I have the following (redacted) meltano.yml config:

version: 1
default_environment: dev
project_id: ...
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-postgres
    variant: meltanolabs
    pip_url: git+https://github.com/MeltanoLabs/tap-postgres.git
    config:
      database: ...
      filter_schemas:
      - ...
      flattening_enabled: false
      host: ...
      ssh_tunnel:
        enable: true
        host: ...
        port: ...
        username: ...
      user: ...

As well as the following .env file

TAP_POSTGRES_PASSWORD='...'
TAP_POSTGRES_SSH_TUNNEL_PRIVATE_KEY='-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY——'

I took the SSH key file content and replaced all new lines with \n, just like in the example test case
https://github.com/MeltanoLabs/tap-postgres/blob/bcc0db15be8184f7d99126fdad7b3a20f7dd91e8/tests/test_ssh_tunnel.py#L14

Furthermore, I checked out the tap code locally and ran my key against the helper method
https://github.com/MeltanoLabs/tap-postgres/blob/bcc0db15be8184f7d99126fdad7b3a20f7dd91e8/tap_postgres/tap.py#L385

to be 100% sure that it's valid and detected as an RSA key type.


That's why I'm highly confused to see this when running meltano config tap-postgres test

2023-09-20T12:05:19.866936Z [info     ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

Plugin configuration is invalid
Catalog discovery failed: command ['/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/bin/tap-postgres', '--config', '/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/run/tap-postgres/tap.598e019d-53d3-4dab-99ff-e2240091d9d6.config.json', '--discover'] returned 1 with stderr:
 Traceback (most recent call last):
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/bin/tap-postgres", line 8, in <module>
    sys.exit(TapPostgres.cli())
             ^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/click/core.py", line 1077, in main
    with self.make_context(prog_name, args, **extra) as ctx:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/click/core.py", line 943, in make_context
    self.parse_args(ctx, args)
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/click/core.py", line 1408, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/click/core.py", line 2400, in handle_parse_result
    value = self.process_value(ctx, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/click/core.py", line 2362, in process_value
    value = self.callback(ctx, self, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 527, in cb_discover
    tap.run_discovery()
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 287, in run_discovery
    catalog_text = self.catalog_json_text
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 307, in catalog_json_text
    return json.dumps(self.catalog_dict, indent=2)
                      ^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/tap_postgres/tap.py", line 475, in catalog_dict
    result["streams"].extend(self.connector.discover_catalog_entries())
                             ^^^^^^^^^^^^^^
  File "/Users/pranas/.pyenv/versions/3.11.3/lib/python3.11/functools.py", line 1001, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/tap_postgres/tap.py", line 378, in connector
    url = self.ssh_tunnel_connect(ssh_config=ssh_config, url=url)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/tap_postgres/tap.py", line 429, in ssh_tunnel_connect
    ssh_private_key=self.guess_key_type(ssh_config["private_key"]),
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/tap_postgres/tap.py", line 414, in guess_key_type
    raise ValueError(errmsg)
ValueError: Could not determine the key type.

ValueError: Could not determine the key type is certainly coming from the guess_key_type method even though I have validated it as explained above. Is this a red herring? How to diagnose further?

Thanks.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing meltano config tap-postgres test with the reported SSH configuration, then read tap_postgres/tap.py around guess_key_type and ssh_tunnel_connect. Compare the input handling with tests/test_ssh_tunnel.py; done means the key-type failure is explained and the reported configuration either works or has regression coverage for the corrected behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
backend, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.