microsoft / microsoft/SPTAG

[4] Cannot open index configure file!

Open
#267 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
5k
Forks
622
Avg merge
8h 37m
Merged PRs (30d)
5

Description

Sptag index build and search works good , Spann index build also works good , but I am facing a problem in the search part of SPANN

This is my SPANN index folder

Screenshot 2022-02-23 at 10 56 22 PM

Index build :
!/content/SPTAG/Release/indexbuilder -c /content/buildconfig.ini -d 784 -v UInt8 -f TXT -i /content/datadata.txt -o /content/spannindex784 -a SPANN

Spann index search

!/content/SPTAG/Release/indexsearcher -x /content/spannindex784/ -d 784 -v UInt8 -f TXT -i /content/dataquery.txt -o /content/outputSearch.txt -k 20

It starts up good but it stops with Cannot open index configure file!, I have not made any changes to the spann index folder , after building index , I provided the same path as it is ,

below are logs when I call spann search . PS:i am using the fashion mnist 784 Euclid dataset

[1] Using AVX2 InstructionSet!
[1] Setting DeletedIDs with value DeletedIDs.bin
[1] Setting DeleteHeadVectors with value false
[1] Setting Dim with value 784
[1] Setting DistCalcMethod with value L2
[1] Setting GenerateTruth with value false
[1] Setting HeadIndexFolder with value HeadIndex
[1] Setting HeadVectorIDs with value SPTAGHeadVectorIDs.bin
[1] Setting HeadVectors with value SPTAGHeadVectors.bin
[1] Setting IndexAlgoType with value BKT
[1] Setting IndexDirectory with value /content/spannindex784
[1] Setting QuantizerFilePath with value quantizer.bin
[1] Setting QueryDelimiter with value |
[1] Setting QueryPath with value /content/dataquery.txt
[1] Setting QuerySize with value -1
[1] Setting QueryType with value DEFAULT
[1] Setting SSDIndex with value SPTAGFullList.bin
[1] Setting SSDIndexFileNum with value 1
[1] Setting TruthPath with value public_query_gt100.bin
[1] Setting TruthType with value DEFAULT
[1] Setting ValueType with value UInt8
[1] Setting VectorDelimiter with value |
[1] Setting VectorPath with value /content/datadata.txt
[1] Setting VectorSize with value 60000
[1] Setting VectorType with value DEFAULT
[1] Setting WarmupDelimiter with value |
[1] Setting WarmupPath with value query.public.10K.u8bin
[1] Setting WarmupSize with value -1
[1] Setting WarmupType with value DEFAULT
[1] Setting AnalyzeOnly with value false
[1] Setting BKTKmeansK with value 32
[1] Setting BKTLambdaFactor with value -1.000000
[1] Setting BKTLeafSize with value 8
[1] Setting CalcStd with value false
[1] Setting Count with value 0
[1] Setting DataBlockSize with value 1048576
[1] Setting DataCapacity with value 2147483647
[1] Setting isExecute with value true
[1] Setting NoOutput with value false
[1] Setting NumberOfThreads with value 45
[1] Setting PrintSizeCount with value true
[1] Setting Ratio with value 0.120000
[1] Setting RecursiveCheckSmallCluster with value true
[1] Setting SamplesNumber with value 1000
[1] Setting SaveBKT with value false
[1] Setting SelectDynamically with value true
[1] Setting SelectHeadType with value BKT
[1] Setting SelectThreshold with value 10
[1] Setting SplitFactor with value 6
[1] Setting SplitMaxTry with value 8
[1] Setting SplitThreshold with value 25
[1] Setting TreeNumber with value 1
[1] Setting AddCEF with value 500
[1] Setting AddCountForRebuild with value 1000
[1] Setting BKTKmeansK with value 32
[1] Setting BKTLambdaFactor with value 100.000000
[1] Setting BKTLeafSize with value 8
[1] Setting BKTNumber with value 1
[1] Setting CEF with value 1000
[1] Setting DataBlockSize with value 1048576
[1] Setting DataCapacity with value 2147483647
[1] Setting DeletePercentageForRefine with value 0.400000
[1] Setting DeleteVectorFilePath with value deletes.bin
[1] Setting DistCalcMethod with value L2
[1] Setting EnableBfs with value 0
[1] Setting EnableRebuild with value 0
[1] Setting GPUGraphType with value 2
[1] Setting GPULeafSize with value 500
[1] Setting GPURefineDepth with value 30
[1] Setting GPURefineSteps with value 0
[1] Setting GraphCEFScale with value 2.000000
[1] Setting GraphFilePath with value graph.bin
[1] Setting GraphNeighborhoodScale with value 2.000000
[1] Setting HashTableExponent with value 2
[1] Setting HeadNumGPUs with value 1
[1] Setting isExecute with value true
[1] Setting MaxCheck with value 16324
[1] Setting MaxCheckForRefineGraph with value 16324
[1] Setting MetaRecordSize with value 10
[1] Setting NeighborhoodSize with value 32
[1] Setting NumberOfInitialDynamicPivots with value 50
[1] Setting NumberOfOtherDynamicPivots with value 4
[1] Setting NumberOfThreads with value 45
[1] Setting NumTopDimensionTpTreeSplit with value 5
[1] Setting RefineIterations with value 3
[1] Setting RNGFactor with value 1.000000
[1] Setting Samples with value 1000
[1] Setting ThresholdOfNumberOfContinuousNoBetterPropagation with value 3
[1] Setting TPTBalanceFactor with value 2
[1] Setting TPTLeafSize with value 2000
[1] Setting TPTNumber with value 32
[1] Setting TreeFilePath with value tree.bin
[1] Setting VectorFilePath with value vectors.bin
[1] Setting Batches with value 1
[1] Setting BuildSsdIndex with value true
[1] Setting DebugBuildInternalResultNum with value 64
[1] Setting EnableADC with value false
[1] Setting GPUSSDLeafSize with value 200
[1] Setting GPUSSDNumTrees with value 100
[1] Setting HashTableExponent with value 2
[1] Setting InternalResultNum with value 64
[1] Setting IOThreadsPerHandler with value 4
[1] Setting isExecute with value true
[1] Setting LogFile with value 
[1] Setting MaxCheck with value 16324
[1] Setting MaxDistRatio with value 8.000000
[1] Setting NumberOfThreads with value 45
[1] Setting NumGPUs with value 1
[1] Setting OutputEmptyReplicaID with value false
[1] Setting PostingPageLimit with value 3
[1] Setting QpsLimit with value 0
[1] Setting QueryCountLimit with value 2147483647
[1] Setting RecallAnalysis with value false
[1] Setting RecallTestSampleNumber with value 100
[1] Setting ReplicaCount with value 8
[1] Setting Rerank with value 0
[1] Setting ResultNum with value 10
[1] Setting RNGFactor with value 1.000000
[1] Setting SearchInternalResultNum with value 32
[1] Setting SearchPostingPageLimit with value 3
[1] Setting SearchResult with value result.txt
[1] Setting TmpDir with value /content/tempfolder
[1] Setting TruthResultNum with value -1
[1] Load Vector (7333,784) Finish!
[1] Load BKT (1,7335) Finish!
[1] Load RNG (7333,32) Finish!
[1] Load DeleteID (7333,1) Finish!
[1] Setting NumberOfThreads with value 45
[1] Setting MaxCheck with value 16324
[1] Setting HashTableExponent with value 2
[1] Finish reading header info, list count 7333, total doc count 60000, dimension 784, list page offset 22.
[1] Big page (>4K): list count 6694, total element count 96954.
[1] Total Element Count: 98081
[1] Page Count Dist: 0 252
[1] Page Count Dist: 1 387
[1] Page Count Dist: 2 369
[1] Page Count Dist: 3 6325
[1] Load MetaIndex(0) Meta(28222)
[4] Error: Failed to load metadata.
[4] Cannot open index configure file!

[4] Cannot open index configure file! , I did not modify any file in the spann index folder yet this error , Please help

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the shown indexbuilder and indexsearcher commands using the SPANN Fashion-MNIST setup. Inspect the generated /content/spannindex784 folder alongside the search log, especially the metadata-loading step and the missing index configuration message. Done means determining why the built index cannot be opened for search and documenting a verified resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.