Failures when running `taskgraph` generations in parallel

Open
#260 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
38/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
python
Domain
build-system, ci-cd

Research direction

Start with taskcluster/ci/test/kind.yml and run taskgraph full -p taskcluster/test/params in parallel to compare it with sequential runs. Trace the KeyError: 'env' through taskgraph/transforms/job/common.py and the related toolchain entry points named in the log. Done means parallel generations complete without the failure and the workaround is no longer needed.

Written by the indexing model from the issue text.

Description

bug

The VPN repo has a task that ran Taskgraph generations in parallel (via taskgraph full -p taskcluster/test/params). For some reason it has failures when running in parallel, that can't be reproduced when running one at a time (I verified this via interactive task). I also was unable to reproduce the failures when running in parallel on my dev machine.

For now, I've worked around the problem by adjusting the task to run each generation one at a time in a for loop.

Here is a snippet from a failing log:

[task 2023-05-31T20:44:24.167Z] Dumping result with parameters from pull-request-untrusted:
[task 2023-05-31T20:44:24.167Z] See '/builds/worker/.cache/taskgraph/log/vcs' for logs
[task 2023-05-31T20:44:24.167Z] concurrent.futures.process._RemoteTraceback: 
[task 2023-05-31T20:44:24.167Z] """
[task 2023-05-31T20:44:24.167Z] Traceback (most recent call last):
[task 2023-05-31T20:44:24.167Z]   File "/usr/lib/python3.10/concurrent/futures/process.py", line 246, in _process_worker
[task 2023-05-31T20:44:24.167Z]     r = call_item.fn(*call_item.args, **call_item.kwargs)
[task 2023-05-31T20:44:24.167Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/main.py", line 146, in format_taskgraph
[task 2023-05-31T20:44:24.167Z]     tg = getattr(tgg, options["graph_attr"])
[task 2023-05-31T20:44:24.167Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/generator.py", line 171, in full_task_graph
[task 2023-05-31T20:44:24.167Z]     return self._run_until("full_task_graph")
[task 2023-05-31T20:44:24.167Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/generator.py", line 423, in _run_until
[task 2023-05-31T20:44:24.167Z]     k, v = next(self._run)
[task 2023-05-31T20:44:24.167Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/generator.py", line 311, in _run
[task 2023-05-31T20:44:24.167Z]     new_tasks = kind.load_tasks(
[task 2023-05-31T20:44:24.167Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/generator.py", line 77, in load_tasks
[task 2023-05-31T20:44:24.167Z]     tasks = [
[task 2023-05-31T20:44:24.167Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/generator.py", line 77, in <listcomp>
[task 2023-05-31T20:44:24.167Z]     tasks = [
[task 2023-05-31T20:44:24.167Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/task.py", line 1317, in check_run_task_caches
[task 2023-05-31T20:44:24.167Z]     for task in tasks:
[task 2023-05-31T20:44:24.167Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/task.py", line 1246, in check_task_dependencies
[task 2023-05-31T20:44:24.167Z]     for task in tasks:
[task 2023-05-31T20:44:24.167Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/task.py", line 1232, in check_task_identifiers
[task 2023-05-31T20:44:24.167Z]     for task in tasks:
[task 2023-05-31T20:44:24.167Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/task.py", line 1213, in chain_of_trust
[task 2023-05-31T20:44:24.167Z]     for task in tasks:
[task 2023-05-31T20:44:24.167Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/task.py", line 1206, in add_github_checks
[task 2023-05-31T20:44:24.167Z]     for task in tasks:
[task 2023-05-31T20:44:24.167Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/task.py", line 1040, in build_task
[task 2023-05-31T20:44:24.168Z]     for task in tasks:
[task 2023-05-31T20:44:24.168Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/task.py", line 1008, in add_index_routes
[task 2023-05-31T20:44:24.168Z]     for task in tasks:
[task 2023-05-31T20:44:24.168Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/task.py", line 936, in process_treeherder_metadata
[task 2023-05-31T20:44:24.168Z]     for task in tasks:
[task 2023-05-31T20:44:24.168Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/task.py", line 898, in validate
[task 2023-05-31T20:44:24.168Z]     for task in tasks:
[task 2023-05-31T20:44:24.168Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/task.py", line 886, in task_name_from_label
[task 2023-05-31T20:44:24.168Z]     for task in tasks:
[task 2023-05-31T20:44:24.168Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/task.py", line 842, in set_defaults
[task 2023-05-31T20:44:24.168Z]     for task in tasks:
[task 2023-05-31T20:44:24.168Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/task.py", line 821, in set_implementation
[task 2023-05-31T20:44:24.168Z]     for task in tasks:
[task 2023-05-31T20:44:24.168Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/cached_tasks.py", line 64, in cache_task
[task 2023-05-31T20:44:24.168Z]     for task in order_tasks(config, tasks):
[task 2023-05-31T20:44:24.168Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/cached_tasks.py", line 22, in order_tasks
[task 2023-05-31T20:44:24.168Z]     pending = deque(tasks)
[task 2023-05-31T20:44:24.168Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/job/__init__.py", line 365, in make_task_description
[task 2023-05-31T20:44:24.168Z]     configure_taskdesc_for_run(
[task 2023-05-31T20:44:24.168Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/job/__init__.py", line 439, in configure_taskdesc_for_run
[task 2023-05-31T20:44:24.168Z]     func(config, job, taskdesc)
[task 2023-05-31T20:44:24.168Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/job/toolchain.py", line 165, in docker_worker_toolchain
[task 2023-05-31T20:44:24.168Z]     common_toolchain(config, job, taskdesc, is_docker=True)
[task 2023-05-31T20:44:24.168Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/job/toolchain.py", line 102, in common_toolchain
[task 2023-05-31T20:44:24.168Z]     docker_worker_add_artifacts(config, job, taskdesc)
[task 2023-05-31T20:44:24.168Z]   File "/builds/worker/.local/lib/python3.10/site-packages/taskgraph/transforms/job/common.py", line 101, in docker_worker_add_artifacts
[task 2023-05-31T20:44:24.168Z]     taskdesc["worker"]["env"]["UPLOAD_DIR"] = path
[task 2023-05-31T20:44:24.168Z] KeyError: 'env'
[task 2023-05-31T20:44:24.168Z] """
[task 2023-05-31T20:44:24.168Z] 
[task 2023-05-31T20:44:24.168Z] The above exception was the direct cause of the following exception:
[task 2023-05-31T20:44:24.168Z] 
[task 2023-05-31T20:44:24.168Z] KeyError: 'env'
Dominant language
Python
Stars
26
Forks
55
Avg merge
2d 6m
Merged PRs (30d)
15

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from taskcluster/taskgraph

All issues in taskcluster/taskgraph

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.