gradle / gradle/guides

Avoid mixing sources advice

Open
#261 0 comments 0 reactions 0 assignees View on GitHub
in:performance
Dominant language
Java
Stars
48
Forks
36
Avg merge
7h 37m
Merged PRs (30d)
8

Description

@eriwen commented on [Wed Dec 05 2018](https://github.com/gradle-guides/performance/issues/26)

Some compilers have a hard time with mixed sources (Java+Groovy or Kotlin, say) in the same source directory.

Having this makes incremental compilation harder to achieve because tools don't know in advance if there is a cyclic type dependency between Groovy => Java => Groovy, say, making things slower.

EDIT: to be clear, we specifically recommend avoiding the following scenarios:

```
src/main/java/
my/package/
MyClass.java
any/package/
OtherClass.kt
```

---

@wolfs commented on [Wed Dec 05 2018](https://github.com/gradle-guides/performance/issues/26#issuecomment-444442762)

For Groovy, if you use two source directories (e.g. `src/main/java` and `src/main/groovy`) at the same time, then Java compilation will run first and then Groovy compilation will run. Java classes cannot reference Groovy classes in this setup.

Adding `.java` and `.groovy` classes into `src/main/groovy` works as well. The Groovy and Java classes can depend on each other. Groovy will create Stubs for the Groovy classes to compile the Java classes and then compile the Groovy classes. The downside is that the Groovy compiler is not incremental.

For Kotlin, when having `java` source directory (e.g. `src/main/java`) and a `kotlin` src directory (e.g. `src/main/kotlin`), then Kotlin compilation is done first and the Kotlin compiler parses the Java classes so it can compile against them. Then java compilation runs against the compiled Kotlin classes. So the Java classes are essentially parsed twice.
Also, changes to Java sources cause Kotlin and Java compilation be out of data, same goes for changes to Kotlin sources.

I don't know if it is possible to add Java classes to `src/main/kotlin`.

---

@CristianGM commented on [Wed Dec 05 2018](https://github.com/gradle-guides/performance/issues/26#issuecomment-444452236)

The common scenario is having both `.java` and `.kotlin` in `src/main/java` and I think is the only scenario you haven't mentioned @wolfs

---

@wolfs commented on [Wed Dec 05 2018](https://github.com/gradle-guides/performance/issues/26#issuecomment-444471705)

I didn't know that even worked...

@CristianGM Does this work out of the box? Or do you need to do something like described here: https://kotlinlang.org/docs/reference/using-gradle.html#android-studio? So it only would affect Android projects.

---

@CristianGM commented on [Wed Dec 05 2018](https://github.com/gradle-guides/performance/issues/26#issuecomment-444481321)

It does work out of the box
> Alternatively, you can put Kotlin classes in the Java source directory, typically located in src/main/java.

But I don't know if it's android only ( as we use the kotlin-android plugin ).

---

@oehme commented on [Wed Dec 05 2018](https://github.com/gradle-guides/performance/issues/26#issuecomment-444617887)

Just here to confirm that it doesn't matter where your put your sources with Kotlin, it'll always do joint compilation.

---

@JLLeitschuh commented on [Wed Dec 05 2018](https://github.com/gradle-guides/performance/issues/26#issuecomment-444624535)

Instead of a documented warning, we just use an internal plugin to prevent these sorts of overlaps. We found that having java in the `kotlin` directory or kotlin in the `java` directory made things like checkstyle and spotbugs not work out of the box.

I'll leave the source code here incase anyone finds it useful.
```kotlin
@CacheableTask
open class ValidateSourceDirectoryTask : DefaultTask() {

@get:Internal
val srcDirs: ConfigurableFileCollection = project.files()

@Suppress("unused")
@get:SkipWhenEmpty
@get:PathSensitive(PathSensitivity.RELATIVE)
@get:InputFiles
val sources: FileTree = srcDirs.asFileTree.matching {
listOf("kt", "kts", "java").forEach {
this.include("**/*.$it")
}
}

@Suppress("LeakingThis")
@get:OutputFile
val reportFile: RegularFileProperty = newOutputFile()

@Suppress("unused")
@TaskAction
fun validate() {
data class ValidatedFile(
val file: File,
val correctFolder: String
) {
val filePath = file.path
}

// Map all source files into just the invalid ones so we can display a list of invalid
// files, instead of failing-fast on the first invalid file
val invalidFiles = srcDirs
.map {
// Map to path validation
when (it.extension) {
"java" -> ValidatedFile(
file = it,
correctFolder = "/java/"
)

"kt", "kts" -> ValidatedFile(
file = it,
correctFolder = "/kotlin/"
)

else -> ValidatedFile(
file = it,
correctFolder = "/resources/"
)
}
}
.filter {
// Filter for invalid files
!it.filePath.contains(it.correctFolder)
}
.filter {
/*
* There is currently no `package-info` equivalent in Kotlin.
* If you have a project that is 100% Kotlin, it doesn't really make sense to
* require a developer to create a `/java/` directory just to hold
* this documentation file.
*/
!it.file.endsWith("package-info.java")
}
.toSet()

if (invalidFiles.isNotEmpty()) {
val errorMessage = invalidFiles.joinToString("\n") {
// Generate error messages
"File ${it.filePath} must be in a ${it.correctFolder} directory."
}

reportFile.get().asFile.writeText(errorMessage)

throw GradleException(errorMessage)
}
}
}

/**
* This plugin validates that all source files are in an appropriate directory for their language.
* This means that Java files are in a /java/ directory, Kotlin files are in a /kotlin/ directory,
* and other files are in a /resources/ directory. When source directories are language-specific,
* the Gradle build system can make stronger assumptions and builds are therefore more performant.
*/
open class ValidateSourceDirectoryPlugin : Plugin {

override fun apply(target: Project) {
target.afterEvaluate {

val sources by lazy {
target.the().sourceSets.flatMap {
it.allSource
}
}

val task =
target.taskRegister("validateSourceDirectories") {
group = LifecycleBasePlugin.VERIFICATION_GROUP
description = "Validates that all files are in their language-appropriate directory."
srcDirs.setFrom(sources)
reportFile.set(target.layout.buildDirectory.file(target.provider {
"reports/validateSourceDirectory/invalidFiles.txt"
}))
}

target.tasks.named(LifecycleBasePlugin.CHECK_TASK_NAME).configure {
dependsOn(task)
}
}
}

private inline fun Project.taskRegister(name: String, noinline configuration: T.() -> Unit) =
this.tasks.register(name, T::class.java, configuration)
}
```

One thing to note is that if you want to be able to call Groovy from Kotlin in a project you need to do something like this gnarly mess:

https://github.com/JLLeitschuh/gradle-kotlin-aspectj-weaver/blob/0c504c477b0993509823735654fdfb801f10495d/plugin/build.gradle.kts#L26-L34

I'm guessing that doing this also borks the up-to date checking all to heck.

If I ever get a chance to re-write this I'd probably make the compilation of the Groovy it's own standalone project.

---

@eriwen commented on [Thu Dec 06 2018](https://github.com/gradle-guides/performance/issues/26#issuecomment-444630295)

I discussed this a bit with @oehme and here's what I learned:

* The Kotlin plugin takes both `src/main/java` and `src/main/kotlin` as an input, which means to avoid expensive joint compilation, one would either have to extract Java sources to a separate project or convert them to Kotlin. This is a low-priority concern because Kotlin only parses the Java files it needs, however...
* Using KAPT makes joint compilation much, much more expensive because it needs to generate stubs and does non-incremental annotation processing.
* Isolating Java from Groovy sources does improve Groovy compile performance.

Thus, modularization is going to be more effective than separating sources, so I filed https://github.com/gradle-guides/performance/issues/27

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the linked performance issue 27, which the discussion identifies as the follow-up on modularization. Review the source-directory examples in the issue and the documented Java, Kotlin, and Groovy compilation behavior. The work is done when the guidance reflects the resolved recommendation rather than the open-ended discussion here.

Written by the indexing model from the issue text.

Assessment

Tech stack
groovy, java, kotlin
Domain
documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.