Expose type dependencies (call graph) for accurate incremental transpiling
- Dominant language
- Java
- Stars
- 1.4k
- Forks
- 161
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
For accurate incremental transpiling, external tooling (Maven, Gradle, etc.) need to determine which files to re-transpile when a given file changes.
The mapping from source file to generated files is needed too (naming conventions are not enough, because it's not unusual for a Java file to contain multiple types, and can even contain several top-level types; and source files aren't even required to follow the common naming convention based on the package and type names) to accurately cleanup the output directory when a source file is deleted.
J2CL also copies source files (`.java` and `.js`) to the output, possibly relocating them, so the mapping from source path to output path is needed as well.
Even better if it could also map to JARs or directories in the `-classpath` for external dependencies, such that if the JAR (or class in the directory) changes the build tool could re-process all files that depend on it (and possibly on all following path entries in the classpath, in case class is now shadowed, or a class that was shadowed no longer is; if it could map to the actual class file, this could be made even more precise, by detecting those shadowing cases). Otherwise, each change to the classpath would mean reprocessing all files. Gradle, for instance, does such processing for its incremental `javac`.
For example, given those two source files:
```java
package p;
class A {
B a() {
return new B(1);
}
}
```
and
```java
package p;
class B {
B(Number n) {}
}
```
If `src/p/B.java` is modified, then we'll want to reprocess both `src/p/B.java` and `src/p/A.java` because `p.B` would have had its API modified in a way that would change `p.A`'s output (e.g. here adding a `B(int)` constructor overload).
Processing those 2 files will result in 8 files being generated or copied:
* `out/p/A.java` (copied from the sources)
* `out/p/A.java.js`
* `out/p/A.impl.java.js`
* `out/p/A.js.map`
* `out/p/B.java` (copied from the sources)
* `out/p/B.java.js`
* `out/p/B.impl.java.js`
* `out/p/B.js.map`
If `src/p/A.java` is deleted, we'll want to delete all the `out/p/A.*` files.
Now if a `src/p/A.native.js` is created, then `src/p/A.java` would have to be reprocessed. And then again if `src/p/A.native.js` is deleted. But because J2CL uses a naming convention here, this does not need to appear in the mapping (correct me if I'm wrong).
If `src/p/B.java` is modified to add an inner class, it would generate 3 new files (`out/p/B$Inner.java.js`, `out/p/B$Inner.impl.java.js`, and `out/p/B$Inner.js.map`). If `src/p/B.java` is then deleted, then all 7 files need to be deleted, and `src/p/A.java` be reprocessed (which would fail, unless it was also modified to remove its dependency on `p.B`).
**Describe the solution you'd like**
Define a stable file format that `J2clCommandLineRunner` could output for use by external (non-Bazel) tooling.
It needs to contain 2 things:
* the list of dependencies between source files (`src/p/A.java` depends on `src/p/B.java`; this could use types as an indirection, e.g. type `p.A` comes from `src/p/A.java` and references type `p.B` that comes from `src/p/B.java`, and `foo.Bar` that comes from `jar:foo.jar!/foo/Bar.class`)
* the mapping back to the source file(s) (there could be a `.java` and a `.native.js`) for each output file
In an incremental use of the transpiler, the build tool would have to merge the generated mapping file with the previously known mapping (and removing entries from that global mapping when a file is deleted).
**Describe alternatives you've considered**
J2CL outputs a source map file (`.js.map`) for each generated file (pair of `.java.js` and `.impl.java.js`), containing the path of the source; but that references the source that J2CL copied to the output, not the actual path to the source (as can be seen in `bazel-bin/third_party/jbox2d.js.zip` vs `bazel-bin/third_party/libjbox2d-src.jar` after a `bazel build //third_party:jbox2d.js.zip`: the `.js.zip` contains a `java/lang/StrictMath.js.map` referencing, using a relative path, `java/lang/StrictMath.java`, but the actual source in the `-src.jar` was `external/org_jbox2d/src/main/java/org/jbox2d/gwtemul/java/lang/StrictMath.java`), because it's targeted at tools that need to access those `.java` files.
**Additional context**
J2CL internally already has all the information; it populates a `LibraryInfo` protobuf that it can serialize to a file (for later use by RTA) but the flag to do so is not exposed to `J2clCommandLineRunner` (probably to keep the `LibraryInfo` as an implementation detail of J2CL). Just like the source map though, the library_info does not contain the actual source path either (modifying `third_party/BUILD` to pass `readable_library_info = True` to `//third_party:jbox2d` generates a `library_info_debug.json` in the `.js.zip` but it does not contain the full `src/main/java/org/jbox2d/gwtemul/java/lang/StrictMath.java` path)
The Kythe indexing metadata includes the information though (I modified `build_defs/internal_do_not_use/j2cl_common.bzl` to unconditionally pass `-generatekytheindexingmetadata`), so J2CL has all the needed information to generate such a mapping file.
Contributor guide
Assessment
This issue has not been assessed yet.