manticoresoftware / manticoresoftware/columnar

Bigint leads to index corruption

Open
#19 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug waiting
Dominant language
C++
Stars
110
Forks
23
Avg merge
3d 5h
Merged PRs (30d)
8

Description

I'm doing an import of a fairly large dataset into Manticore with columnar. The table has several number fields and when these are set as type integer the import finishes without a hitch. However, if i change these to bigint the import crashes after a few minutes. Not instantly, but consistently after about 1M inserts and without any SQL errors but the log says:

malloc(): unsorted double linked list corrupted
Crash!!! Handling signal 6
malloc(): invalid size (unsorted)
Crash!!! Handling signal 11

The bigint's are negative and would overflow 32bit, so using integer to start with was a mistake from my side. I can then not continue the import and the table is corrupted. When I drop the table and recreate it I get the message:

Create schema index err: Error 1064 (42000): error adding table 'TABLENAME': directory is not empty: /var/lib/manticore/TABLENAME

Meaning I have to clear the whole manticore directory and start over.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with a large columnar import using negative bigint values, then inspect the crash output and the table directory behavior after dropping and recreating the table. The fix is complete when the import no longer crashes or corrupts the table and the table can be recreated without manually clearing the whole Manticore directory.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, sql
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.