apache / apache/maven-indexer

Missing sha1 digests from nexus-maven-repository-index.gz

Open
#692 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
74
Forks
69
Avg merge
14h 8m
Merged PRs (30d)
6

Description

### Affected version

latest

### Bug description

There are some sha1 digests missing from [nexus-maven-repository-index.gz](https://repo.maven.apache.org/maven2/.index/nexus-maven-repository-index.gz)
One such example is: [92b6631659ba35ca09e44874d3eb936edfeee532](https://central.sonatype.com/search?q=1%3A92b6631659ba35ca09e44874d3eb936edfeee532) related to [jakarta.validation-api](https://repo1.maven.org/maven2/jakarta/validation/jakarta.validation-api/3.0.2/)
```
$ curl https://search.maven.org/solrsearch/select\?q\="1:92b6631659ba35ca09e44874d3eb936edfeee532"

{"responseHeader":{"status":0,"QTime":111,"params":{"q":"1:92b6631659ba35ca09e44874d3eb936edfeee532","core":"","indent":"off","fl":"id,g,a,v,p,ec,timestamp,tags","start":"","sort":"score desc,timestamp desc,g asc,a asc,v desc","rows":"20","wt":"json","version":"2.2"}},"response":{"numFound":1,"start":0,"docs":[{"id":"jakarta.validation:jakarta.validation-api:3.0.2","g":"jakarta.validation","a":"jakarta.validation-api","v":"3.0.2","p":"jar","timestamp":1653588099000,"ec":["-sources.jar",".pom","-javadoc.jar",".jar"],"tags":["jakarta","bean","validation"]}]}}%
```

Looking at the index with Luke I get the following results when searching for `g:jakarta.validation v:3.0.2`:
```
17136615 8.715791 u=jakarta.validation|jakarta.validation-api|3.0.2|sources|jar.asc.sha512;
17136616 8.715791 1=b1581d1ef516be94c473c0f4d97e59e394c70150; u=jakarta.validation|jakarta.validation-api|3.0.2|sources|jar;
17136617 8.715791 1=eb8bb58bbce6761e2e8095f48f383da117fb109c; u=jakarta.validation|jakarta.validation-api|3.0.2|javadoc|jar;
62019267 8.715791 u=jakarta.validation|jakarta.validation-api|3.0.2|NA|pom.asc.sha512;
62019268 8.715791 u=jakarta.validation|jakarta.validation-api|3.0.2|sources|jar.asc.sha256;
62019269 8.715791 u=jakarta.validation|jakarta.validation-api|3.0.2|javadoc|jar.asc.sha512;
62019270 8.715791 u=jakarta.validation|jakarta.validation-api|3.0.2|javadoc|jar.asc.sha256;
```

Here, I can only see the sha1sums for `sources.jar` and `javadoc.jar`, but not for the [main jar](https://repo1.maven.org/maven2/jakarta/validation/jakarta.validation-api/3.0.2/jakarta.validation-api-3.0.2.jar.sha1) with digest `92b6631659ba35ca09e44874d3eb936edfeee532`
Image

It may be related [to this other IndexSearcher issue](https://github.com/apache/maven-indexer/issues/579).

What I also tried was running the latest version of the indexer on the following directory structure:
```
$ tree jakarta
jakarta
└── validation
└── jakarta.validation-api
└── 3.0.2
├── jakarta.validation-api-3.0.2-javadoc.jar
├── jakarta.validation-api-3.0.2-javadoc.jar.asc
├── jakarta.validation-api-3.0.2-javadoc.jar.asc.md5
├── jakarta.validation-api-3.0.2-javadoc.jar.asc.sha1
├── jakarta.validation-api-3.0.2-javadoc.jar.asc.sha256
├── jakarta.validation-api-3.0.2-javadoc.jar.asc.sha512
├── jakarta.validation-api-3.0.2-javadoc.jar.md5
├── jakarta.validation-api-3.0.2-javadoc.jar.sha1
├── jakarta.validation-api-3.0.2-sources.jar
├── jakarta.validation-api-3.0.2-sources.jar.asc
├── jakarta.validation-api-3.0.2-sources.jar.asc.md5
├── jakarta.validation-api-3.0.2-sources.jar.asc.sha1
├── jakarta.validation-api-3.0.2-sources.jar.asc.sha256
├── jakarta.validation-api-3.0.2-sources.jar.asc.sha512
├── jakarta.validation-api-3.0.2-sources.jar.md5
├── jakarta.validation-api-3.0.2-sources.jar.sha1
├── jakarta.validation-api-3.0.2.jar
├── jakarta.validation-api-3.0.2.jar.asc
├── jakarta.validation-api-3.0.2.jar.asc.md5
├── jakarta.validation-api-3.0.2.jar.asc.sha1
├── jakarta.validation-api-3.0.2.jar.asc.sha256
├── jakarta.validation-api-3.0.2.jar.asc.sha512
├── jakarta.validation-api-3.0.2.jar.md5
├── jakarta.validation-api-3.0.2.jar.sha1
├── jakarta.validation-api-3.0.2.pom
├── jakarta.validation-api-3.0.2.pom.asc
├── jakarta.validation-api-3.0.2.pom.asc.md5
├── jakarta.validation-api-3.0.2.pom.asc.sha1
├── jakarta.validation-api-3.0.2.pom.asc.sha256
├── jakarta.validation-api-3.0.2.pom.asc.sha512
├── jakarta.validation-api-3.0.2.pom.md5
└── jakarta.validation-api-3.0.2.pom.sha1

4 directories, 32 files

$ java -jar indexer-cli/target/indexer-cli-7.1.7-SNAPSHOT-cli.jar -X -i /tmp/jakarta-validation-download/.index -d /tmp/jakarta-validation-download/.index -r /tmp/jakarta-validation-download
Version: 7.1.7-SNAPSHOT
Repository Folder: /tmp/jakarta-validation-download
Index Folder: /tmp/jakarta-validation-download/.index
Output Folder: /tmp/jakarta-validation-download/.index
Repository name: .index
Indexers: [min, jarContent]
Will not create checksum files.
Will create baseline file.
Scanning started
1 jakarta/validation/jakarta.validation-api/3.0.2/jakarta.validation-api-3.0.2.pom
2 jakarta/validation/jakarta.validation-api/3.0.2/jakarta.validation-api-3.0.2.pom
3 jakarta/validation/jakarta.validation-api/3.0.2/jakarta.validation-api-3.0.2.pom
4 jakarta/validation/jakarta.validation-api/3.0.2/jakarta.validation-api-3.0.2.pom
Artifacts added: 4
Artifacts deleted: 0
[main] INFO org.apache.maven.index.packer.DefaultIndexPacker - Unable to read properties file, will force index regeneration
Total time: 0 sec
Final memory: 29M/260M
```
And I got the following error when unpacking the index:
```
$ java -jar ~/work/maven-indexer/indexer-cli/target/indexer-cli-7.1.7-SNAPSHOT-cli.jar -u
Index Folder: /private/tmp/jakarta-validation-download/.index
Output Folder: /private/tmp/jakarta-validation-download/.index
java.lang.IllegalArgumentException: cannot change field "m" from index options=NONE to inconsistent index options=DOCS_AND_FREQS_AND_POSITIONS
at org.apache.lucene.index.FieldInfo.verifySameIndexOptions(FieldInfo.java:273)
at org.apache.lucene.index.FieldInfos$FieldNumbers.verifySameSchema(FieldInfos.java:567)
at org.apache.lucene.index.FieldInfos$FieldNumbers.addOrGet(FieldInfos.java:476)
at org.apache.lucene.index.IndexWriter.addIndexes(IndexWriter.java:3074)
at org.apache.maven.index.updater.IndexDataReader.readIndexMT(IndexDataReader.java:268)
at org.apache.maven.index.updater.IndexDataReader.readIndex(IndexDataReader.java:132)
at org.apache.maven.index.updater.DefaultIndexUpdater.unpackIndexData(DefaultIndexUpdater.java:344)
at org.apache.maven.index.updater.DefaultIndexUpdater.unpackIndexData(DefaultIndexUpdater.java:324)
at org.apache.maven.index.cli.NexusIndexerCli.unpack(NexusIndexerCli.java:500)
at org.apache.maven.index.cli.NexusIndexerCli.execute(NexusIndexerCli.java:157)
at org.apache.maven.index.cli.NexusIndexerCli.main(NexusIndexerCli.java:116)
```

I'm curious to know what the search API does in order to find this digest that isn't present in the central maven index and what I can do in order to accurately get all the sha1sums identifying the java packages.

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the reported jakarta.validation index with the indexer-cli commands and inspect IndexDataReader, DefaultIndexUpdater, and NexusIndexerCli around index creation and unpacking. Compare the generated index entries with the missing main-jar SHA-1 and the related IndexSearcher issue 579. Done means the digest is represented accurately and the generated index can be unpacked without the reported Lucene field-schema error.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.