Allow fetch tasks to be generated by embedded entries in other tasks

Open
#186 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
python

Research direction

Review the two linked prototype comparisons and the TransformSequence discussion, including fetch_only_job_transfoms. Clarify the desired embedded-fetch behavior and formalize a complete approach; done means the behavior is no longer hardcoded and transform generation is defined well enough to support the stated examples.

Written by the indexing model from the issue text.

Description

For simple use cases, embedded fetch definitions into tasks could be much nicer than the indirection of having a separate kind for them. It can also be nice to embed them where only one entry in a kind needs them -- it puts the fetch with the thing that actually needs it, which makes it more obvious what's going on, and lowers maintenance burden a small amount.

I've put together a prototype for how this could work in these patches:

It's a bit hacky at the moment, but it does prove that we can generate unrelated tasks (eg: a fetch) while processing tasks from another kind (eg: build). This may seem very strange, and it certainly breaks new ground in taskgraph. When I look at things through the eyes of someone building a project (in this case MozillaVPN), the tasks I really care about are build, test, etc. -- things that produce artifacts that are of value to developers or users. fetch, on the other hand, is an implementation detail. There are certainly good reasons for them existing in a separate kind at times (most obviously, when a single fetch is used by many downstream tasks) -- but that shouldn't be strictly necessary.

Obviously the linked patches are not remotely landable. There's hardcoding that has to be fixed, and most importantly, the generation of the TransformsSequence would need to be formalized (especially the fetch_only_job_transfoms part, where it's taking only part of the job transforms -- I have some ideas on how to clean that up).

We can also just close this as undesirable if it's going too far or a bad idea for some reason.

Dominant language
Python
Stars
26
Forks
55
Avg merge
2d 6m
Merged PRs (30d)
15

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from taskcluster/taskgraph

All issues in taskcluster/taskgraph

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.