CodeForPhilly / CodeForPhilly/cfp-data-pipeline
github tap failed
- Dominant language
- Python
- Stars
- 7
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
@dherbst, I'm not seeing a lot in the log--any thoughts on what might be causing it. Does the bot user need a specific kind of access to each of these repos (besides read?)
Log below
```
*** Reading local file: /usr/local/airflow/logs/meltano/github-to-postgres/2021-02-15T00:00:00+00:00/2.log
[2021-02-16 22:04:00,790] {taskinstance.py:826} INFO - Dependencies all met for
[2021-02-16 22:04:00,819] {taskinstance.py:826} INFO - Dependencies all met for
[2021-02-16 22:04:00,819] {taskinstance.py:1017} INFO -
--------------------------------------------------------------------------------
[2021-02-16 22:04:00,819] {taskinstance.py:1018} INFO - Starting attempt 2 of 2
[2021-02-16 22:04:00,820] {taskinstance.py:1019} INFO -
--------------------------------------------------------------------------------
[2021-02-16 22:04:00,852] {taskinstance.py:1038} INFO - Executing on 2021-02-15T00:00:00+00:00
[2021-02-16 22:04:00,857] {standard_task_runner.py:51} INFO - Started process 3811 to run task
[2021-02-16 22:04:00,873] {standard_task_runner.py:75} INFO - Running: ['airflow', 'tasks', 'run', 'meltano', 'github-to-postgres', '2021-02-15T00:00:00+00:00', '--job-id', '12', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/meltano.py', '--cfg-path', '/tmp/tmp35wd3sgw']
[2021-02-16 22:04:00,878] {standard_task_runner.py:76} INFO - Job 12: Subtask github-to-postgres
[2021-02-16 22:04:00,979] {logging_mixin.py:103} INFO - Running on host 6e6178ea0d91
[2021-02-16 22:04:01,066] {taskinstance.py:1230} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_EMAIL=mchow@codeforphilly.org
AIRFLOW_CTX_DAG_OWNER=Michael Chow
AIRFLOW_CTX_DAG_ID=meltano
AIRFLOW_CTX_TASK_ID=github-to-postgres
AIRFLOW_CTX_EXECUTION_DATE=2021-02-15T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=scheduled__2021-02-15T00:00:00+00:00
[2021-02-16 22:04:01,067] {bash.py:135} INFO - Tmp dir root location:
/tmp
[2021-02-16 22:04:01,068] {bash.py:158} INFO - Running command: /usr/local/venv/meltano/bin/meltano elt tap-github target-postgres-github --job_id=github-to-postgres
[2021-02-16 22:04:01,082] {bash.py:169} INFO - Output:
[2021-02-16 22:04:05,023] {bash.py:173} INFO - meltano | Running extract & load...
[2021-02-16 22:04:05,365] {bash.py:173} INFO - meltano | No state was found, complete import.
[2021-02-16 22:04:06,285] {bash.py:173} INFO - tap-github | INFO Starting sync of repository: CodeforPhilly/After_School_Wiki
[2021-02-16 22:04:06,382] {bash.py:173} INFO - tap-github | INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.08558869361877441, "tags": {"endpoint": "milestones", "status": "failed"}}
[2021-02-16 22:04:06,382] {bash.py:173} INFO - tap-github | INFO METRIC: {"type": "counter", "metric": "record_count", "value": 0, "tags": {"endpoint": "issue_milestones"}}
[2021-02-16 22:04:06,383] {bash.py:173} INFO - tap-github | CRITICAL {"message":"Not Found","documentation_url":"https://docs.github.com/rest/reference/issues#list-milestones"}
[2021-02-16 22:04:06,385] {bash.py:173} INFO - tap-github | Traceback (most recent call last):
[2021-02-16 22:04:06,385] {bash.py:173} INFO - tap-github | File "/usr/local/meltano/cfp-pipeline/.meltano/extractors/tap-github/venv/bin/tap-github", line 8, in
[2021-02-16 22:04:06,385] {bash.py:173} INFO - tap-github | sys.exit(main())
[2021-02-16 22:04:06,385] {bash.py:173} INFO - tap-github | File "/usr/local/meltano/cfp-pipeline/.meltano/extractors/tap-github/venv/lib/python3.8/site-packages/singer/utils.py", line 229, in wrapped
[2021-02-16 22:04:06,385] {bash.py:173} INFO - tap-github | return fnc(*args, **kwargs)
[2021-02-16 22:04:06,385] {bash.py:173} INFO - tap-github | File "/usr/local/meltano/cfp-pipeline/.meltano/extractors/tap-github/venv/lib/python3.8/site-packages/tap_github/__init__.py", line 1388, in main
[2021-02-16 22:04:06,385] {bash.py:173} INFO - tap-github | do_sync(args.config, args.state, catalog)
[2021-02-16 22:04:06,385] {bash.py:173} INFO - tap-github | File "/usr/local/meltano/cfp-pipeline/.meltano/extractors/tap-github/venv/lib/python3.8/site-packages/tap_github/__init__.py", line 1357, in do_sync
[2021-02-16 22:04:06,385] {bash.py:173} INFO - tap-github | state = sync_func(stream_schema, repo, state, mdata)
[2021-02-16 22:04:06,385] {bash.py:173} INFO - tap-github | File "/usr/local/meltano/cfp-pipeline/.meltano/extractors/tap-github/venv/lib/python3.8/site-packages/tap_github/__init__.py", line 492, in get_all_issue_milestones
[2021-02-16 22:04:06,385] {bash.py:173} INFO - tap-github | for response in authed_get_all_pages(
[2021-02-16 22:04:06,385] {bash.py:173} INFO - tap-github | File "/usr/local/meltano/cfp-pipeline/.meltano/extractors/tap-github/venv/lib/python3.8/site-packages/tap_github/__init__.py", line 148, in authed_get_all_pages
[2021-02-16 22:04:06,385] {bash.py:173} INFO - tap-github | r = authed_get(source, url, headers)
[2021-02-16 22:04:06,386] {bash.py:173} INFO - tap-github | File "/usr/local/meltano/cfp-pipeline/.meltano/extractors/tap-github/venv/lib/python3.8/site-packages/backoff/_sync.py", line 94, in retry
[2021-02-16 22:04:06,386] {bash.py:173} INFO - tap-github | ret = target(*args, **kwargs)
[2021-02-16 22:04:06,386] {bash.py:173} INFO - tap-github | File "/usr/local/meltano/cfp-pipeline/.meltano/extractors/tap-github/venv/lib/python3.8/site-packages/tap_github/__init__.py", line 140, in authed_get
[2021-02-16 22:04:06,386] {bash.py:173} INFO - tap-github | raise NotFoundException(resp.text)
[2021-02-16 22:04:06,386] {bash.py:173} INFO - tap-github | tap_github.NotFoundException: {"message":"Not Found","documentation_url":"https://docs.github.com/rest/reference/issues#list-milestones"}
[2021-02-16 22:04:06,397] {bash.py:173} INFO - target-postgres-github | INFO PostgresTarget created with established connection: `user=postgres password=xxx dbname=datawarehouse host=postgres port=5432 sslmode=prefer sslcert=~/.postgresql/postgresql.crt sslkey=~/.postgresql/postgresql.key sslrootcert=~/.postgresql/root.crt sslcrl=~/.postgresql/root.crl`, PostgreSQL schema: `tap_github`
[2021-02-16 22:04:06,401] {bash.py:173} INFO - target-postgres-github | INFO Sending version information to singer.io. To disable sending anonymous usage data, set the config parameter "disable_collection" to true
[2021-02-16 22:04:06,556] {bash.py:173} INFO - meltano | Extraction failed (1): tap_github.NotFoundException: {"message":"Not Found","documentation_url":"https://docs.github.com/rest/reference/issues#list-milestones"}
[2021-02-16 22:04:06,556] {bash.py:173} INFO - meltano | ELT could not be completed: Extractor failed
[2021-02-16 22:04:06,589] {bash.py:173} INFO - ELT could not be completed: Extractor failed
[2021-02-16 22:04:06,960] {bash.py:177} INFO - Command exited with return code 1
[2021-02-16 22:04:06,978] {taskinstance.py:1396} ERROR - Bash command failed. The command returned a non-zero exit code.
Traceback (most recent call last):
File "/opt/venv/reticulate/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1086, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/opt/venv/reticulate/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1260, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/opt/venv/reticulate/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1300, in _execute_task
result = task_copy.execute(context=context)
File "/opt/venv/reticulate/lib/python3.8/site-packages/airflow/operators/bash.py", line 180, in execute
raise AirflowException('Bash command failed. The command returned a non-zero exit code.')
airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code.
[2021-02-16 22:04:06,981] {taskinstance.py:1433} INFO - Marking task as FAILED. dag_id=meltano, task_id=github-to-postgres, execution_date=20210215T000000, start_date=20210216T220400, end_date=20210216T220406
[2021-02-16 22:04:07,033] {local_task_job.py:118} INFO - Task exited with return code 1
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.