apache / apache/lucene

LowercaseAsciiCompression should return false when it's unable to compress [LUCENE-10551]

Open
#11,587 12 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

```java
Failed to commit..
java.lang.IllegalStateException: 10 <> 5 cion1cion_desarrollociones_oraclecionesnaturacionesnatura2tedppsa-integrationdemotiontion cloud gen2tion instance - dev1tion instance - testtion-devbtion-instancetion-prdtion-promerication-qation064533tion535217tion697401tion761348tion892818tion_matrationcauto_simmonsintgic_testtioncloudprodictioncloudservicetiongateway10tioninstance-jtsundatamartprd??o
        at org.apache.lucene.util.compress.LowercaseAsciiCompression.compress(LowercaseAsciiCompression.java:115)
        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.writeBlock(BlockTreeTermsWriter.java:834)
        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.writeBlocks(BlockTreeTermsWriter.java:628)
        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.pushTerm(BlockTreeTermsWriter.java:947)
        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:912)
        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:318)
        at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:170)
        at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:120)
        at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:267)
        at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:350)
        at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:476)
        at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:656)
        at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3364)
        at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3770)
        at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3728)
     
```

```java
key=och-live--WorkResource.renewAssignmentToken.ResourceTime[namespace=workflow, resourceGroup=workflow-service-overlay]{availabilityDomain=iad-ad-1, domainId=och-live, host=workflow-service-overlay-01341.node.ad1.us-ashburn-1})
java.lang.IllegalStateException: 29 <> 16 analytics-platform-test/koala/cluster-tool:1.0-20220310151438.492,mesh_istio_examples-bookinfo-details-v1:1.16.2mesh_istio_examples-bookinfo-reviews-v3:1.16.2oce-clamav:1.0.219oce-tesseract:1.0.7oce-traefik:2.5.1oci-opensearch:1.2.4.8.103oda-digital-assistant-control-plane-train-pool-workflow-v6:22.02.14oke-coresvcs-k8s-dns-dnsmasq-nanny-amd64@sha256:41aa9160ceeaf712369ddb660d02e5ec06d1679965e6930351967c8cf5ed62d4oke-coresvcs-k8s-dns-kube-dns-amd64@sha256:2cf34b04106974952996c6ef1313f165ce65b4ad68a3051f51b1b8f91ba5f838oke-coresvcs-k8s-dns-sidecar-amd64@sha256:8a82c7288725cb4de9c7cd8d5a78279208e379f35751539b406077f9a3163dcdoke-coresvcs-node-problem-detector@sha256:9d54df11804a862c54276648702a45a6a0027a9d930a86becd69c34cc84bf510oke-coresvcs-oke-fluentd-lumberjack@sha256:5f3f10b187eb804ce4e84bc3672de1cf318c0f793f00dac01cd7da8beea8f269oke-etcd-operator@sha256:4353a2e5ef02bb0f6b046a8d6219b1af359a2c1141c358ff110e395f29d0bfc8oke-oke-hyperkube-amd64@sha256:3c734f46099400507f938090eb9a874338fa25cde425ac9409df4c885759752foke-public-busybox@sha256:4cee1979ba0bf7db9fc5d28fb7b798ca69ae95a47c5fecf46327720df4ff352doke-public-coredns@sha256:86f8cfc74497f04e181ab2e1d26d2fd8bd46c4b33ce24b55620efcdfcb214670oke-public-coredns@sha256:8cd974302f1f6108f6f31312f8181ae723b514e2022089cdcc3db10666c49228oke-public-etcd@sha256:b751e459bc2a8f079f6730dd8462671b253c7c8b0d0eb47c67888d5091c6bb77oke-public-etcd@sha256:d6a76200a6e9103681bc2cf7fefbcada0dd9372d52cf8964178d846b89959d14oke-public-etcd@sha256:fa056479342b45479ac74c58176ddad43687d5fc295375d705808f9dfb48439aoke-public-kube-proxy@sha256:93b2da69d03413671606e22294c59a69fe404088a5f6e74d6394a8641fdb899boke-public-tiller@sha256:c2eb6e580123622e1bc0ff3becae3a3a71ac36c98a2786d780590197839175e5osms/opcbuild-osms-agent-proxy-java:0.4.0-129rosms/opcbuild-osms-agent-proxy-nginx:0.4.0-129rosms/opcbuild-osms-ingestion-cert:0.4.0-129rscs-lcm/drift-detector:227scs-lcm/salt-state-sync:242streaming-alpine:30.10.183streaming-kafka:30.10.183vision-service-document-classification:1.1.55vision-service-image-classification:1.4.52
        at org.apache.lucene.util.compress.LowercaseAsciiCompression.compress(LowercaseAsciiCompression.java:115)
        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.writeBlock(BlockTreeTermsWriter.java:834)
        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.writeBlocks(BlockTreeTermsWriter.java:628)
        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.pushTerm(BlockTreeTermsWriter.java:947)
        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:912)
        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:318)
        at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:170)
        at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:120)
        at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:267)
        at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:350)
        at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:476)
        at org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:390)
        at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:436)
        at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1471)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1757)
        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1400)
```

When addDocument and commit we are seeing IllegalStateException from 

LowercaseAsciiCompression when push term into blocks.

Should have a better exception message or change to return false

---
Migrated from [LUCENE-10551](https://issues.apache.org/jira/browse/LUCENE-10551) by Peixin Li, updated May 13 2022
Environment:
```
Lucene version 8.11.1
```

Attachments: [LUCENE-10551-test.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-10551/LUCENE-10551-test.patch)
Pull requests: https://github.com/apache/lucene/pull/858, https://github.com/apache/lucene/pull/858

Contributor guide

Open the contributing guide

Research direction

Start at org.apache.lucene.util.compress.LowercaseAsciiCompression.compress, especially line 115, and review the attached LUCENE-10551-test.patch. Follow its callers in BlockTreeTermsWriter and run the relevant compression or indexing tests. Done means the reported terms no longer trigger the current IllegalStateException and the resulting behavior matches the issue's agreed resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.