dlt-hub / dlt-hub/dlt-init-openapi

Generator fails for missing project_dir

Open
#126 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
39
Forks
8
PR merge metrics
No merged PRs in 30d

Description

I'm encountering some blocking issues when trying to generate the API source for posthog.

So just running the cli with no configuration whatsoever I get this output:
```console
$ dlt-init-openapi --url https://app.posthog.com/api/schema/
2024-07-02 18:13:50.181 | SUCCESS | dlt_init_openapi.cli:_init_command_wrapped:91 - Starting dlt openapi generator
2024-07-02 18:13:50.181 | INFO | dlt_init_openapi.utils.update_rest_api:update_rest_api:14 - Syncing rest_api verified source
2024-07-02 18:13:50.182 | INFO | dlt_init_openapi.utils.update_rest_api:update_rest_api:18 - rest_api verified source already present
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.10/site-packages/dlt_init_openapi │
│ /cli/__init__.py:58 in init │
│ │
│ 55 ) -> None: │
│ 56 │ """Generate a new dlt pipeline""" │
│ 57 │ │
│ ❱ 58 │ _init_command_wrapped( │
│ 59 │ │ source=source, │
│ 60 │ │ url=url, │
│ 61 │ │ path=path, │
│ │
│ ╭──────────────────────────── locals ────────────────────────────╮ │
│ │ allow_openapi_2 = False │ │
│ │ config_path = None │ │
│ │ global_limit = 0 │ │
│ │ interactive = True │ │
│ │ log_level = 20 │ │
│ │ output_path = None │ │
│ │ path = None │ │
│ │ source = None │ │
│ │ update_rest_api_source = False │ │
│ │ url = 'https://app.posthog.com/api/schema/' │ │
│ │ version = None │ │
│ ╰────────────────────────────────────────────────────────────────╯ │
│ │
│ /home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.10/site-packages/dlt/common/runti │
│ me/telemetry.py:97 in _wrap │
│ │
│ 94 │ │ │ │ return f(*f_args, **f_kwargs) │
│ 95 │ │ │ # some commands we track after, where we can pass the success │
│ 96 │ │ │ try: │
│ ❱ 97 │ │ │ │ rv = f(*f_args, **f_kwargs) │
│ 98 │ │ │ │ # if decorated function returns int, 0 is a success - used to track dlt │
│ 99 │ │ │ │ if isinstance(rv, int): │
│ 100 │ │ │ │ │ success = rv == 0 │
│ │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ _track = .decorator.._wrap.._track at │ │
│ │ 0x7fe9796e1f30> │ │
│ │ args = ('source', 'url', 'path') │ │
│ │ bound_args = │ │
│ │ category = 'command' │ │
│ │ command = 'init-openapi' │ │
│ │ f = │ │
│ │ f_args = () │ │
│ │ f_kwargs = { │ │
│ │ │ 'source': None, │ │
│ │ │ 'url': 'https://app.posthog.com/api/schema/', │ │
│ │ │ 'path': None, │ │
│ │ │ 'output_path': None, │ │
│ │ │ 'config_path': None, │ │
│ │ │ 'interactive': True, │ │
│ │ │ 'log_level': 20, │ │
│ │ │ 'global_limit': 0, │ │
│ │ │ 'update_rest_api_source': False, │ │
│ │ │ 'allow_openapi_2': False │ │
│ │ } │ │
│ │ props = { │ │
│ │ │ 'source': None, │ │
│ │ │ 'url': 'https://app.posthog.com/api/schema/', │ │
│ │ │ 'path': None, │ │
│ │ │ 'elapsed': 0.003656625747680664, │ │
│ │ │ 'success': False, │ │
│ │ │ 'event_category': 'command', │ │
│ │ │ 'event_name': 'init-openapi' │ │
│ │ } │ │
│ │ sig = None> │ │
│ │ start_ts = 1719936830.178589 │ │
│ │ track_before = False │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /home/drnick/repos/analytics/42next-modeling/.venv/lib/python3.10/site-packages/dlt_init_openapi │
│ /cli/__init__.py:119 in _init_command_wrapped │
│ │
│ 116 │ │ │ }, │
│ 117 │ │ ) │
│ 118 │ │ │
│ ❱ 119 │ │ if config.project_dir.exists(): │
│ 120 │ │ │ if not interactive: │
│ 121 │ │ │ │ logger.info("Non interactive mode selected, overwriting existing source. │
│ 122 │ │ │ elif not questionary.confirm( │
│ │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ allow_openapi_2 = False │ │
│ │ config = Config( │ │
│ │ │ output_path=None, │ │
│ │ │ project_name=None, │ │
│ │ │ package_name=None, │ │
│ │ │ post_hooks=[ │ │
│ │ │ │ 'autoflake -i -r --remove-all-unused-imports │ │
│ │ --remove-unused-variables .', │ │
│ │ │ │ 'isort --float-to-top .', │ │
│ │ │ │ 'black .' │ │
│ │ │ ], │ │
│ │ │ include_methods=['get'], │ │
│ │ │ fallback_openapi_title='openapi', │ │
│ │ │ project_folder_suffix='_pipeline', │ │
│ │ │ pipeline_file_suffix='_pipeline.py', │ │
│ │ │ dataset_name_suffix='_data', │ │
│ │ │ endpoint_filter=, │ │
│ │ │ name_resources_by_operation=False, │ │
│ │ │ │ │
│ │ renderer_class='dlt_init_openapi.renderer.default.DefaultRenderer', │ │
│ │ │ │ │
│ │ detector_class='dlt_init_openapi.detector.default.DefaultDetector', │ │
│ │ │ global_limit=0, │ │
│ │ │ required_parameter_default_value='FILL_ME_IN', │ │
│ │ │ unrequired_parameter_default_value='OPTIONAL_CONFIG', │ │
│ │ │ allow_openapi_2=False, │ │
│ │ │ project_dir=None, │ │
│ │ │ pipeline_file_name=None, │ │
│ │ │ spec_url='https://app.posthog.com/api/schema/', │ │
│ │ │ spec_path=None │ │
│ │ ) │ │
│ │ config_path = None │ │
│ │ create_new_client = │ │
│ │ global_limit = 0 │ │
│ │ interactive = True │ │
│ │ log_level = 20 │ │
│ │ output_path = None │ │
│ │ path = None │ │
│ │ source = None │ │
│ │ update_rest_api_source = False │ │
│ │ url = 'https://app.posthog.com/api/schema/' │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'NoneType' object has no attribute 'exists'
```
So it seems that `config.project_dir` is `None`. Seeing as there is no way to configure it without passing a `config.yaml`, I then created one which is very simple
```yaml
project_name: posthog
project_folder_suffix: test
```
It fails with an cryptic error
```console
$ dlt-init-openapi --url https://app.posthog.com/api/schema/ --config config.yaml
2024-07-02 18:37:39.111 | SUCCESS | dlt_init_openapi.cli:_init_command_wrapped:91 - Starting dlt openapi generator
2024-07-02 18:37:39.112 | INFO | dlt_init_openapi.utils.update_rest_api:update_rest_api:14 - Syncing rest_api verified source
2024-07-02 18:37:39.112 | INFO | dlt_init_openapi.utils.update_rest_api:update_rest_api:18 - rest_api verified source already present
Usage: dlt-init-openapi [OPTIONS] [SOURCE]
Try 'dlt-init-openapi --help' for help.
╭─ Error ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Invalid value: Unable to parse config │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
```
So then I was debugging the cli and it turns out that the real error is here
```python
def load_from_path(path: Path, *args: Any, **kwargs: Any) -> "Config":
"""Creates a Config from provided JSON or YAML file and sets a bunch of globals from it"""
mime = mimetypes.guess_type(path.absolute().as_uri(), strict=True)[0]
if mime == "application/json":
config_data = json.loads(path.read_text())
else:
config_data = yaml.safe_load(path.read_text())
config = Config(**config_data, **kwargs) ## This is the real error
return config
```
`TypeError("dlt_init_openapi.config.Config() got multiple values for keyword argument 'project_name'")`

This is because the keyword is set to None in `config_data` and in the `kwargs`

```python
config_data = {'project_folder_suffix': 'test', 'project_name': 'posthog'}
kwargs = {'project_name': None, 'package_name': None, 'output_path': None, 'endpoint_filter': , 'global_limit': 0, 'spec_url': 'https://app.posthog.com/api/schema/', 'spec_path': None, 'allow_openapi_2': False}
```

and I can't seem to find a workaround without rewriting the cli. Pydantic (or is it python) doesn't support multiple keywords.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.