INRIA / INRIA/spoon

bug: Spoon fails to index modules when loaded from ZIP

Open
#4,137 0 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Java
Stars
2k
Forks
392
Avg merge
11h 24m
Merged PRs (30d)
36

Description

## Problem statement
It would be kind of funny if it wasn't so sad, but after #4052 indexing no longer works correctly with a ZIP as input source... Indexing from the file system works though :upside_down_face:

## More elaborate problem statement
My JavadocAPI refuses to find Javadoc for quite a few entries in JDK 16 when it is indexed from a ZIP file. The element is there but it finds no comment. This only happens if I include a substantial part of the JDK in the zip.

I would love to test [my MWE from the other bug](https://github.com/INRIA/spoon/issues/4052#issuecomment-886087548) but that crashes Spoon during model building:
```
==== Building spoon model ====
Starting phase COMPILE
Phase COMPILE done! Discovered Classes: 49
Starting phase COMMENT
Phase COMMENT done! Discovered Classes: 49
Starting phase MODEL
Exception in thread "main" java.lang.NullPointerException: Cannot invoke "org.eclipse.jdt.internal.compiler.lookup.AnnotationBinding.getAnnotationType()" because "annotation" is null
at spoon.support.compiler.jdt.JDTTreeBuilderQuery.hasAnnotationWithType(JDTTreeBuilderQuery.java:178)
at spoon.support.compiler.jdt.ParentExiter.scanCtElement(ParentExiter.java:156)
at spoon.reflect.visitor.CtInheritanceScanner.visitCtField(CtInheritanceScanner.java:593)
at spoon.support.reflect.declaration.CtFieldImpl.accept(CtFieldImpl.java:53)
at spoon.reflect.visitor.CtInheritanceScanner.scan(CtInheritanceScanner.java:181)
at spoon.support.compiler.jdt.ContextBuilder.exit(ContextBuilder.java:132)
at spoon.support.compiler.jdt.JDTTreeBuilder.endVisit(JDTTreeBuilder.java:640)
at org.eclipse.jdt.internal.compiler.ast.SingleMemberAnnotation.traverse(SingleMemberAnnotation.java:76)
at org.eclipse.jdt.internal.compiler.ast.FieldDeclaration.traverse(FieldDeclaration.java:375)
at org.eclipse.jdt.internal.compiler.ast.TypeDeclaration.traverse(TypeDeclaration.java:1699)
at org.eclipse.jdt.internal.compiler.ast.CompilationUnitDeclaration.traverse(CompilationUnitDeclaration.java:827)
at org.eclipse.jdt.internal.compiler.ast.CompilationUnitDeclaration.traverse(CompilationUnitDeclaration.java:788)
at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.traverseUnitDeclaration(JDTBasedSpoonCompiler.java:480)
at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.lambda$buildModel$0(JDTBasedSpoonCompiler.java:437)
at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.forEachCompilationUnit(JDTBasedSpoonCompiler.java:464)
at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildModel(JDTBasedSpoonCompiler.java:435)
at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildUnitsAndModel(JDTBasedSpoonCompiler.java:372)
at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildSources(JDTBasedSpoonCompiler.java:335)
at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:116)
at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:99)
at spoon.Launcher.buildModel(Launcher.java:781)
at de.ialistannen.javadocapi.indexing.Indexer.main(Indexer.java:62)
```
The MWE works when I unzip it however...

CC: @andrewbwogi

----
EDIT:
A lead could be this
![image](https://user-images.githubusercontent.com/20284688/131226478-55808671-1561-4d31-ba99-c669e6e5efa7.png)

result of the JDT "Main" class trying to extract the module info by parsing the compilation unit. This works when the file is named correctly without the random temporary file numbers.

I "fixed" this locally by doing:
```diff
diff --git a/src/main/java/spoon/compiler/builder/SourceOptions.java b/src/main/java/spoon/compiler/builder/SourceOptions.java
index f602d34e..b9e2d27c 100644
--- a/src/main/java/spoon/compiler/builder/SourceOptions.java
+++ b/src/main/java/spoon/compiler/builder/SourceOptions.java
@@ -7,14 +7,15 @@
*/
package spoon.compiler.builder;

-import org.apache.commons.io.IOUtils;
-import spoon.compiler.SpoonFile;
-
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
import java.util.Arrays;
import java.util.List;
+import org.apache.commons.io.IOUtils;
+import spoon.compiler.SpoonFile;

public class SourceOptions> extends Options {
public SourceOptions() {
@@ -45,21 +46,46 @@ public class SourceOptions> extends Options {
args.add(".");
return myself;
}
- for (SpoonFile source : sources) {
- if (source.isActualFile()) {
- args.add(source.toString());
- } else {
+ try {
+ Path path = Files.createTempDirectory("spoon-container");
+
+ Runtime.getRuntime().addShutdownHook(new Thread(() -> {
try {
- File file = File.createTempFile(source.getName(), ".java");
- file.deleteOnExit();
- try (FileOutputStream fileOutputStream = new FileOutputStream(file)) {
- IOUtils.copy(source.getContent(), fileOutputStream);
- }
- args.add(file.toString());
+ Files.walk(path).sorted(Comparator.reverseOrder()).forEach(
+ path1 -> {
+ try {
+ Files.deleteIfExists(path1);
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ });
} catch (IOException e) {
- throw new RuntimeException(e.getMessage(), e);
+ e.printStackTrace();
+ }
+ }));
+
+ for (SpoonFile source : sources) {
+ if (source.isActualFile()) {
+ args.add(source.toString());
+ } else {
+ try {
+ String name = source.getName();
+ if (name.startsWith("/")) {
+ name = name.substring(1);
+ }
+ File file = path.resolve(name).toFile();
+ file.getParentFile().mkdirs();
+ try (FileOutputStream fileOutputStream = new FileOutputStream(file)) {
+ IOUtils.copy(source.getContent(), fileOutputStream);
+ }
+ args.add(file.toString());
+ } catch (IOException e) {
+ throw new RuntimeException(e.getMessage(), e);
+ }
}
}
+ } catch (IOException e) {
+ e.printStackTrace();
}
return myself;
}
```
Which names the files correctly after extracting them from the ZIP.

Contributor guide

Open the contributing guide

Research direction

Start with src/main/java/spoon/compiler/builder/SourceOptions.java and the JDTBasedSpoonCompiler stack-trace path. Reproduce model building with the ZIP input, then inspect how extracted source names affect module parsing and annotation handling. Done means ZIP-loaded modules index correctly without the reported missing comments or NullPointerException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
compilers, devtools
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.