manticoresoftware / manticoresoftware/columnar
Bigint leads to index corruption
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 110
- Forks
- 23
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 8
Description
I'm doing an import of a fairly large dataset into Manticore with columnar. The table has several number fields and when these are set as type integer the import finishes without a hitch. However, if i change these to bigint the import crashes after a few minutes. Not instantly, but consistently after about 1M inserts and without any SQL errors but the log says:
malloc(): unsorted double linked list corrupted
Crash!!! Handling signal 6
malloc(): invalid size (unsorted)
Crash!!! Handling signal 11
The bigint's are negative and would overflow 32bit, so using integer to start with was a mistake from my side. I can then not continue the import and the table is corrupted. When I drop the table and recreate it I get the message:
Create schema index err: Error 1064 (42000): error adding table 'TABLENAME': directory is not empty: /var/lib/manticore/TABLENAME
Meaning I have to clear the whole manticore directory and start over.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with a large columnar import using negative bigint values, then inspect the crash output and the table directory behavior after dropping and recreating the table. The fix is complete when the import no longer crashes or corrupts the table and the table can be recreated without manually clearing the whole Manticore directory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, sql
- Domain
- database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100