How to make it easier to wire binaries and tasks together
- Dominant language
- No language data
- Stars
- 94
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
```
class Foo extends DefaultTask {
@OutputDirectory DirectoryProperty outputDirectory = newOutputDirectory()
}
class Other extends DefaultTask {
@InputFile RegularFileProperty inputFile = newInputFile()
@InputDirectory DirectoryProperty inputDirectory = newInputDirectory()
}
// This works
otherTask.inputDirectory = fooTask.outputDirectory
// This does not
otherTask.inputFile = fooTask.outputDirectory.file("something")
// Now with lazy task configuration:
// This does not and the type is wrong/weird (TaskProvider -> Provider)
otherTask.inputDirectory = fooTaskProvider.map { it.outputDirectory }
// This does not and it's a little less weird (TaskProvider -> Provider)
otherTask.inputDirectory = fooTaskProvider.map { it.outputDirectory.get() }
```
In some of our native integration tests, we do things like:
```
task buildDebug {
dependsOn application.binaries.get { !it.optimized }.map { it.executableFile }
}
```
This is similar to the examples above. We take a `BinaryProvider` and transform it to a `Provider>`. This works because we end up adding the _value_ of the provider as a dependency which happens to be a `RegularFileProperty` under the covers. This gets unwrapped again and the dependency is maintained.
This PR fixes some of the holes by following dependencies through transform operations: https://github.com/gradle/gradle/pull/5504
The same sort of thing needs to be considered for "combining" providers, locking providers and collection providers. I don't think the PR can be merged as-is because it exposes a problem in the adhoc handling of Provider dependencies now. With the PR changes, the gradle/gradle build runs into this kind of error:
https://builds.gradle.org/viewLog.html?buildId=13103116&buildTypeId=Gradle_Check_Gradleception&tab=buildLog&_focus=3028
This is coming out of:
https://github.com/gradle/kotlin-dsl/blob/44868ed7dcd6a1bad53bd5746ab1159e4b3b84a0/plugins-experiments/src/main/kotlin/org/gradle/kotlin/dsl/experiments/plugins/GradleKotlinDslKtlintConventionPlugin.kt#L62-L66
We create a `Provider` and it's put into a `FileCollection` and then we unwrap it trying to find dependencies. I went back and forth but I couldn't find a combination of changes that would keep everything working _and_ allow us to follow dependencies through `map`. I think I need to take a step back and come up with a list of scenarios vs relying on our existing coverage through integration tests.
This branch builds on top of the Provider changes and converts the native plugins to use `createLater` https://github.com/gradle/gradle/commits/sg/lazy/native-plugins
In all of the native plugins, we currently:
- Create all "variant identities"
- Create all buildable binaries
- From each binary, create tasks
- Configure each task
- Wire task and binary together (binary -> compile task, binary outputs -> compile task outputs, compile task toolchain -> binary tool chain)
Binaries are "read only", deriving values from the component, the variant identity or from an underlying task. For many properties, the task is the configuration entry point.
For lazy task configuration, the process can be the same overall; however, when we need to wire together the binary and the task, we need to lazily attach the task's properties to the binary.
e.g., instead of eagerly creating the compile task and assigning the Provider of its objectFileDir to the binary's objectsDir
https://github.com/gradle/gradle/commit/1a882dedc5f08b90670eb1ef627fcdd2d93722c7#diff-febb5d1178f294372dd78a4ebdb93d9bL111
We can lazily convert the TaskProvider into a Provider:
https://github.com/gradle/gradle/commit/1a882dedc5f08b90670eb1ef627fcdd2d93722c7#diff-febb5d1178f294372dd78a4ebdb93d9bR122
But this requires that dependencies travel through the transformation/map and the code is more complicated now.
Some examples...
Pseudo code today:
```
class CompileTask {
DirectoryProperty objectsDirectory
}
class Binary {
Provider objectsDirectory
Provider compileTask
}
compileTask = tasks.create("compile", CompileTask) {
objectsDirectory = buildDirectory.dir("objects/" + binary.name)
}
((InternalBinary)binary).compileTask.set(compileTask)
((InternalBinary)binary).objectsDirectory.set(compileTask.objectsDirectory)
```
Pros:
- Changes to `compileTask.objectsDirectory` is automatically reflected through `binary.objectsDirectory`. There's only a single way to modify this value.
Cons:
- Hard to write a rule that would affect the `objectsDirectory` property for a particular set of binaries without going through the task.
- Hard to add steps between the built in tasks as the "binary" level, you need to know all about the tasks.
Mitigations:
- Introduce `TaskProvider` for tasks related to a binary. Configuration for the task is more directly connected to the binary then.
Pseudo code lazy:
```
class CompileTask {
DirectoryProperty objectsDirectory
}
class Binary {
Provider objectsDirectory
Provider compileTask
}
compileTask = tasks.createLater("compile", CompileTask) {
objectsDirectory = buildDirectory.dir("objects/" + binary.name)
}
((InternalBinary)binary).compileTask.set(compileTask)
((InternalBinary)binary).objectsDirectory.set(compileTask.map { it.objectsDirectory.get() })
```
Pros:
- As above.
Cons:
- As above.
- `map` is easy to get wrong. Requires other changes to make Providers better.
As an alternative not in the branch, pseudo code lazy with configurable binary and task:
```
class CompileTask {
DirectoryProperty objectsDirectory
}
class Binary {
DirectoryProperty objectsDirectory
Property compileTask
}
binary.objectsDirectory.set(buildDirectory.dir("objects/" + binary.name))
compileTask = tasks.createLater("compile", CompileTask) {
objectsDirectory = binary.objectsDirectory
}
binary.compileTask.set(compileTask)
```
Pros:
- Binary becomes the natural place for configuration.
- No complicated mapping between binary and task properties
Mehs:
- This is a lot like the software model (for better or worse).
Cons:
- How do we know `binary.objectsDirectory` is built by `compileTask`? This is fundamentally missing.
- May require changes to Provider still? (need to spike this)
- Very easy to break the chain if the wrong thing is configured/referenced. Everything needs to use the binary only and never configure/reference the task's properties directly.
- If the binary has all of the task configuration anyways, binary configuration becomes more expensive and defeats the purpose of configuring fewer things.
Mitigations:
- We could make the task less configurable (e.g., using constructor args).
Ultimately, I think we need to do something like the last option (configurable binary, less configurable task), but I'm not convinced we need to do that now.
@adammurdoch WDYT? ☝️There are a lot of things going on here. Does it seem like a reasonable first step to make `Provider`s work in a consistent way WRT task dependencies? I think we can punt on trying to make the native plugins lazy this week.
Contributor guide
Research direction
Start with Provider dependency handling and PR #5504, then inspect plugins-experiments/src/main/kotlin/org/gradle/kotlin/dsl/experiments/plugins/GradleKotlinDslKtlintConventionPlugin.kt around lines 62-66. Review the native plugin createLater branch and run the Gradle build or relevant integration coverage. Done requires an agreed set of provider, transform, locking, collection, and task-dependency scenarios.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, kotlin, swift
- Domain
- build-system, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100