microsoft / microsoft/SPTAG

Segmentation fault when building SIFT1B

Open
#416 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
5k
Forks
622
Avg merge
8h 37m
Merged PRs (30d)
5

Description

When I constructing the index for SIFT1B, the process consistently encounters a Segmentation fault after approximately 20 hours of operation.
image

I suspect the recurring Segmentation fault might be related to memory constraints, but I have allocated 512GB of memory and 32 cores in PVE for this container. Interestingly, even when using the learning subset of SIFT1B, which contains only 100 million vectors, the same issue occurs.

Here is my build script:

[Base]
ValueType=UInt8
DistCalcMethod=L2
IndexAlgoType=BKT
Dim=128
VectorPath=/home/gary/Code/DiskANN/build/data/bigann/bigann_learn.bbin
VectorType=DEFAULT
QueryPath==/home/gary/Code/DiskANN/build/data/bigann/bigann_query.bbin
QueryType=DEFAULT
WarmupPath==/home/gary/Code/DiskANN/build/data/bigann/bigann_query.bbin
WarmupType=DEFAULT
TruthPath==/home/gary/Code/DiskANN/build/data/bigann/bigann_gt.ibin
TruthType=DEFAULT
IndexDirectory=/home/gary/Code/SPTAG/gary_test/sift1b_index_learn

[SelectHead]
isExecute=true
TreeNumber=1
BKTKmeansK=32
BKTLeafSize=8
SamplesNumber=1000
SaveBKT=false
SelectThreshold=10
SplitFactor=6
SplitThreshold=25
Ratio=0.12
NumberOfThreads=32
BKTLambdaFactor=1.0

[BuildHead]
isExecute=true
NeighborhoodSize=32
TPTNumber=32
TPTLeafSize=2000
MaxCheck=16324
MaxCheckForRefineGraph=16324
RefineIterations=3
NumberOfThreads=45
BKTLambdaFactor=-1.0

[BuildSSDIndex]
isExecute=true
BuildSsdIndex=true
InternalResultNum=64
ReplicaCount=8
PostingPageLimit=3
NumberOfThreads=45
MaxCheck=16324
TmpDir=/tmp/

[SearchSSDIndex]
isExecute=true
BuildSsdIndex=false
InternalResultNum=96
NumberOfThreads=1
HashTableExponent=4
ResultNum=10
MaxCheck=1024
MaxDistRatio=8.0
SearchPostingPageLimit=3

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the provided build script and compare the SelectHead, BuildHead, and BuildSSDIndex settings, including their thread counts, for the SIFT1B learning-subset build. Reproduce the long-running build and capture where the segmentation fault occurs; done means identifying the failure cause and documenting a reliable fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.