Segmentation fault when building SIFT1B
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 5k
- Forks
- 622
- Avg merge
- 8h 37m
- Merged PRs (30d)
- 5
Description
When I constructing the index for SIFT1B, the process consistently encounters a Segmentation fault after approximately 20 hours of operation.
I suspect the recurring Segmentation fault might be related to memory constraints, but I have allocated 512GB of memory and 32 cores in PVE for this container. Interestingly, even when using the learning subset of SIFT1B, which contains only 100 million vectors, the same issue occurs.
Here is my build script:
[Base]
ValueType=UInt8
DistCalcMethod=L2
IndexAlgoType=BKT
Dim=128
VectorPath=/home/gary/Code/DiskANN/build/data/bigann/bigann_learn.bbin
VectorType=DEFAULT
QueryPath==/home/gary/Code/DiskANN/build/data/bigann/bigann_query.bbin
QueryType=DEFAULT
WarmupPath==/home/gary/Code/DiskANN/build/data/bigann/bigann_query.bbin
WarmupType=DEFAULT
TruthPath==/home/gary/Code/DiskANN/build/data/bigann/bigann_gt.ibin
TruthType=DEFAULT
IndexDirectory=/home/gary/Code/SPTAG/gary_test/sift1b_index_learn
[SelectHead]
isExecute=true
TreeNumber=1
BKTKmeansK=32
BKTLeafSize=8
SamplesNumber=1000
SaveBKT=false
SelectThreshold=10
SplitFactor=6
SplitThreshold=25
Ratio=0.12
NumberOfThreads=32
BKTLambdaFactor=1.0
[BuildHead]
isExecute=true
NeighborhoodSize=32
TPTNumber=32
TPTLeafSize=2000
MaxCheck=16324
MaxCheckForRefineGraph=16324
RefineIterations=3
NumberOfThreads=45
BKTLambdaFactor=-1.0
[BuildSSDIndex]
isExecute=true
BuildSsdIndex=true
InternalResultNum=64
ReplicaCount=8
PostingPageLimit=3
NumberOfThreads=45
MaxCheck=16324
TmpDir=/tmp/
[SearchSSDIndex]
isExecute=true
BuildSsdIndex=false
InternalResultNum=96
NumberOfThreads=1
HashTableExponent=4
ResultNum=10
MaxCheck=1024
MaxDistRatio=8.0
SearchPostingPageLimit=3
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the provided build script and compare the SelectHead, BuildHead, and BuildSSDIndex settings, including their thread counts, for the SIFT1B learning-subset build. Reproduce the long-running build and capture where the segmentation fault occurs; done means identifying the failure cause and documenting a reliable fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100