influxdata / influxdata/influxdb
Writing many columns corrupts database
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Steps to reproduce:__
I use the below golang test procedure. The influxdb3 service is not responding and corrupted then.
```
var buf string = "{\"Vcf1\":3162,\"Vcf10\":3162,\"Vcf100\":3161,\"Vcf101\":3161,\"Vcf102\":3162,\"Vcf103\":3161,\"Vcf104\":3159,\"Vcf105\":3161,\"Vcf106\":3162,\"Vcf107\":3161,\"Vcf108\":3163,\"Vcf109\":3162,\"Vcf11\":3160,\"Vcf110\":3162,\"Vcf111\":3162,\"Vcf112\":3161,\"Vcf113\":3161,\"Vcf114\":3163,\"Vcf115\":3162,\"Vcf116\":3162,\"Vcf117\":3162,\"Vcf118\":3162,\"Vcf119\":3164,\"Vcf12\":3162,\"Vcf120\":3161,\"Vcf121\":3160,\"Vcf122\":3160,\"Vcf123\":3163,\"Vcf124\":3164,\"Vcf125\":3161,\"Vcf126\":3161,\"Vcf127\":3163,\"Vcf128\":3163,\"Vcf129\":3163,\"Vcf13\":3161,\"Vcf130\":3162,\"Vcf131\":3161,\"Vcf132\":3161,\"Vcf133\":3161,\"Vcf134\":3161,\"Vcf135\":3163,\"Vcf136\":3162,\"Vcf137\":3160,\"Vcf138\":3162,\"Vcf139\":3163,\"Vcf14\":3167,\"Vcf140\":3164,\"Vcf141\":3165,\"Vcf142\":3161,\"Vcf143\":3161,\"Vcf144\":3159,\"Vcf145\":3162,\"Vcf146\":3160,\"Vcf147\":3163,\"Vcf148\":3163,\"Vcf149\":3160,\"Vcf15\":3166,\"Vcf150\":3161,\"Vcf151\":3163,\"Vcf152\":3161,\"Vcf153\":3162,\"Vcf154\":3160,\"Vcf155\":3162,\"Vcf156\":3161,\"Vcf157\":3162,\"Vcf158\":3159,\"Vcf159\":3159,\"Vcf16\":3164,\"Vcf160\":3156,\"Vcf161\":3160,\"Vcf162\":3161,\"Vcf163\":3159,\"Vcf164\":3159,\"Vcf165\":3160,\"Vcf166\":3159,\"Vcf167\":3161,\"Vcf168\":3160,\"Vcf169\":3158,\"Vcf17\":3163,\"Vcf170\":3159,\"Vcf171\":3162,\"Vcf172\":3161,\"Vcf173\":3163,\"Vcf174\":3159,\"Vcf175\":3158,\"Vcf176\":3163,\"Vcf177\":3163,\"Vcf178\":3160,\"Vcf179\":3159,\"Vcf18\":3163,\"Vcf180\":3160,\"Vcf181\":3161,\"Vcf182\":3164,\"Vcf183\":3159,\"Vcf184\":3163,\"Vcf185\":3160,\"Vcf186\":3164,\"Vcf187\":3160,\"Vcf188\":3159,\"Vcf189\":3161,\"Vcf19\":3164,\"Vcf190\":3163,\"Vcf191\":3160,\"Vcf192\":3159,\"Vcf193\":3159,\"Vcf194\":3159,\"Vcf195\":3163,\"Vcf196\":3160,\"Vcf197\":3157,\"Vcf198\":3160,\"Vcf199\":3160,\"Vcf2\":3162,\"Vcf20\":3165,\"Vcf200\":3163,\"Vcf201\":3161,\"Vcf202\":3159,\"Vcf203\":3160,\"Vcf204\":3162,\"Vcf205\":3162,\"Vcf206\":3160,\"Vcf207\":3159,\"Vcf208\":3161,\"Vcf209\":3163,\"Vcf21\":3162,\"Vcf210\":3163,\"Vcf211\":3163,\"Vcf212\":3163,\"Vcf213\":3163,\"Vcf214\":3163,\"Vcf215\":3162,\"Vcf216\":3164,\"Vcf217\":3162,\"Vcf218\":3163,\"Vcf219\":3164,\"Vcf22\":3163,\"Vcf220\":3164,\"Vcf221\":3163,\"Vcf222\":3164,\"Vcf223\":3164,\"Vcf224\":3163,\"Vcf225\":3163,\"Vcf226\":3163,\"Vcf227\":3163,\"Vcf228\":3168,\"Vcf229\":3165,\"Vcf23\":3164,\"Vcf230\":3163,\"Vcf231\":3164,\"Vcf232\":3164,\"Vcf233\":3163,\"Vcf234\":3163,\"Vcf235\":3164,\"Vcf236\":3161,\"Vcf237\":3162,\"Vcf238\":3162,\"Vcf239\":3163,\"Vcf24\":3164,\"Vcf240\":3164,\"Vcf241\":3160,\"Vcf242\":3162,\"Vcf243\":3163,\"Vcf244\":3164,\"Vcf245\":3165,\"Vcf246\":3161,\"Vcf247\":3161,\"Vcf248\":3158,\"Vcf249\":3160,\"Vcf25\":3164,\"Vcf250\":3159,\"Vcf251\":3162,\"Vcf252\":3161,\"Vcf253\":3159,\"Vcf254\":3161,\"Vcf255\":3162,\"Vcf256\":3160,\"Vcf257\":3161,\"Vcf258\":3158,\"Vcf259\":3160,\"Vcf26\":3164,\"Vcf260\":3161,\"Vcf261\":3158,\"Vcf262\":3160,\"Vcf263\":3162,\"Vcf264\":3162,\"Vcf265\":3160,\"Vcf266\":3158,\"Vcf267\":3160,\"Vcf268\":3162,\"Vcf269\":3160,\"Vcf27\":3162,\"Vcf270\":3161,\"Vcf271\":3159,\"Vcf272\":3159,\"Vcf273\":3161,\"Vcf274\":3163,\"Vcf275\":3163,\"Vcf276\":3163,\"Vcf277\":3163,\"Vcf278\":3161,\"Vcf279\":3163,\"Vcf28\":3160,\"Vcf280\":3164,\"Vcf281\":3162,\"Vcf282\":3161,\"Vcf283\":3163,\"Vcf284\":3161,\"Vcf285\":3164,\"Vcf286\":3162,\"Vcf287\":3163,\"Vcf288\":3162,\"Vcf289\":3160,\"Vcf29\":3162,\"Vcf290\":3161,\"Vcf291\":3162,\"Vcf292\":3164,\"Vcf293\":3164,\"Vcf294\":3160,\"Vcf295\":3160,\"Vcf296\":3161,\"Vcf297\":3162,\"Vcf298\":3164,\"Vcf299\":3162,\"Vcf3\":3161,\"Vcf30\":3162,\"Vcf300\":3163,\"Vcf301\":3163,\"Vcf302\":3161,\"Vcf303\":3162,\"Vcf304\":3160,\"Vcf305\":3162,\"Vcf306\":3162,\"Vcf307\":3163,\"Vcf308\":3161,\"Vcf309\":3161,\"Vcf31\":3164,\"Vcf310\":3160,\"Vcf311\":3164,\"Vcf312\":3164,\"Vcf313\":3161,\"Vcf314\":3161,\"Vcf315\":3162,\"Vcf316\":3160,\"Vcf317\":3162,\"Vcf318\":3161,\"Vcf319\":3161,\"Vcf32\":3163,\"Vcf320\":3162,\"Vcf321\":3162,\"Vcf322\":3161,\"Vcf323\":3161,\"Vcf324\":3161,\"Vcf325\":3163,\"Vcf326\":3165,\"Vcf327\":3165,\"Vcf328\":3162,\"Vcf329\":3163,\"Vcf33\":3159,\"Vcf330\":3162,\"Vcf331\":3163,\"Vcf332\":3164,\"Vcf333\":3162,\"Vcf334\":3163,\"Vcf335\":3164,\"Vcf336\":3162,\"Vcf337\":3164,\"Vcf338\":3163,\"Vcf339\":3162,\"Vcf34\":3163,\"Vcf340\":3161,\"Vcf341\":3165,\"Vcf342\":3164,\"Vcf343\":3165,\"Vcf344\":3163,\"Vcf345\":3161,\"Vcf346\":3163,\"Vcf347\":3164,\"Vcf348\":3163,\"Vcf349\":3162,\"Vcf35\":3164,\"Vcf350\":3162,\"Vcf351\":3163,\"Vcf352\":3160,\"Vcf353\":3159,\"Vcf354\":3159,\"Vcf355\":3159,\"Vcf356\":3160,\"Vcf357\":3161,\"Vcf358\":3156,\"Vcf359\":3159,\"Vcf36\":3163,\"Vcf360\":3161,\"Vcf361\":3160,\"Vcf362\":3159,\"Vcf363\":3160,\"Vcf364\":3159,\"Vcf365\":3161,\"Vcf366\":3160,\"Vcf367\":3157,\"Vcf368\":3157,\"Vcf369\":3159,\"Vcf37\":3161,\"Vcf370\":3158,\"Vcf371\":3159,\"Vcf372\":3157,\"Vcf373\":3158,\"Vcf374\":3159,\"Vcf375\":3160,\"Vcf376\":3161,\"Vcf377\":3158,\"Vcf378\":3163,\"Vcf379\":3164,\"Vcf38\":3161,\"Vcf380\":3162,\"Vcf381\":3163,\"Vcf382\":3161,\"Vcf383\":3164,\"Vcf384\":3164,\"Vcf385\":3162,\"Vcf386\":3161,\"Vcf387\":3163,\"Vcf388\":3163,\"Vcf389\":3164,\"Vcf39\":3163,\"Vcf390\":3163,\"Vcf391\":3164,\"Vcf392\":3163,\"Vcf393\":3161,\"Vcf394\":3160,\"Vcf395\":3162,\"Vcf396\":3164,\"Vcf397\":3163,\"Vcf398\":3161,\"Vcf399\":3161,\"Vcf4\":3163,\"Vcf40\":3160,\"Vcf400\":3161,\"Vcf401\":3164,\"Vcf402\":3164,\"Vcf403\":3160,\"Vcf404\":3161,\"Vcf405\":3157,\"Vcf406\":3159,\"Vcf407\":3158,\"Vcf408\":3159,\"Vcf409\":3161,\"Vcf41\":3161,\"Vcf410\":3159,\"Vcf411\":3157,\"Vcf412\":3160,\"Vcf413\":3159,\"Vcf414\":3162,\"Vcf415\":3160,\"Vcf416\":3157,\"Vcf42\":3161,\"Vcf43\":3162,\"Vcf44\":3159,\"Vcf45\":3158,\"Vcf46\":3163,\"Vcf47\":3161,\"Vcf48\":3157,\"Vcf49\":3158,\"Vcf5\":3160,\"Vcf50\":3160,\"Vcf51\":3162,\"Vcf52\":3161,\"Vcf53\":3159,\"Vcf54\":3158,\"Vcf55\":3162,\"Vcf56\":3160,\"Vcf57\":3161,\"Vcf58\":3159,\"Vcf59\":3159,\"Vcf6\":3161,\"Vcf60\":3161,\"Vcf61\":3162,\"Vcf62\":3161,\"Vcf63\":3161,\"Vcf64\":3159,\"Vcf65\":3159,\"Vcf66\":3164,\"Vcf67\":3166,\"Vcf68\":3164,\"Vcf69\":3163,\"Vcf7\":3161,\"Vcf70\":3163,\"Vcf71\":3163,\"Vcf72\":3164,\"Vcf73\":3162,\"Vcf74\":3162,\"Vcf75\":3165,\"Vcf76\":3164,\"Vcf77\":3165,\"Vcf78\":3164,\"Vcf79\":3164,\"Vcf8\":3161,\"Vcf80\":3163,\"Vcf81\":3160,\"Vcf82\":3161,\"Vcf83\":3159,\"Vcf84\":3162,\"Vcf85\":3164,\"Vcf86\":3162,\"Vcf87\":3161,\"Vcf88\":3161,\"Vcf89\":3163,\"Vcf9\":3162,\"Vcf90\":3164,\"Vcf91\":3161,\"Vcf92\":3162,\"Vcf93\":3161,\"Vcf94\":3160,\"Vcf95\":3159,\"Vcf96\":3163,\"Vcf97\":3164,\"Vcf98\":3161,\"Vcf99\":3159}"
func Test_WriteLineProtocol(t *testing.T) {
//client := make([]Client, 300)
var client Client
//for i := 0; i < 300; i++ {
var err error
client, err = NewClient(&opts)
if err != nil {
t.Error(err)
return
}
//}
data := make(map[string]any)
if err := json.Unmarshal([]byte(buf), &data); err != nil {
t.Error(err)
}
var wg sync.WaitGroup
wg.Add(300)
ts := time.Now()
for i := 0; i < 300; i++ {
go func() {
for j := 0; j < 200; j++ {
errWrite := client.WriteLineProtocol(
context.Background(),
"lineproto2",
map[string]string{
"tag1": "value1",
},
data,
time.Now(),
)
if errWrite != nil {
t.Error(errWrite)
}
}
wg.Done()
}()
}
wg.Wait()
t.Logf("time passed: %v", time.Since(ts))
}
```
When I restart the influxdb3 but without any write load, the cpu rate of influxdb3 is dramattically high like below.
```
top - 15:56:06 up 8 min, 1 user, load average: 25.55, 27.95, 15.40
Tasks: 130 total, 1 running, 128 sleeping, 0 stopped, 1 zombie
%Cpu(s): 1.8 us, 87.9 sy, 0.0 ni, 0.0 id, 9.5 wa, 0.0 hi, 0.9 si, 0.0 st
MiB Mem : 1708.3 total, 905.3 free, 845.2 used, 151.9 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 863.1 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
44 root 20 0 0 0 0 S 90.3 0.0 1:44.78 kswapd0
5939 influxd+ 20 0 0 0 0 Z 53.9 0.0 0:07.29 influxdb3
```
The data file directory can be shared if needed.
__Expected behaviour:__
Influxdb3 core should support concurrent write.
__Actual behaviour:__
concurrent writing seems to cause the database corrupted and not self-restoring.
__Environment info:__
I just download the influxdb3 prebuild binary from official website.
__Config:__
INFLUXDB3_OBJECT_STORE=file
INFLUXDB3_DB_DIR=/opt/influxdb3/data
INFLUXDB3_NODE_IDENTIFIER_PREFIX=my-host
LOG_FILTER=info
INFLUXDB3_MAX_HTTP_REQUEST_SIZE=20971520
INFLUXDB3_WAL_FLUSH_INTERVAL=1s
LOG_DESTINATION=stdout
LOG_FORMAT=full
__Logs:__
Include snippet of errors in logs or stack traces here.
Sometimes you can get useful information by running the program with the `RUST_BACKTRACE=full` environment variable.
Finally, the IOx server has a `-vv` for verbose logging.
Contributor guide
Research direction
Start by reproducing the issue with the provided Go Test_WriteLineProtocol test and its concurrent Client.WriteLineProtocol calls against the configured influxdb3 binary. Capture logs with verbose output or RUST_BACKTRACE=full and compare behavior after restart. Done means concurrent writes no longer corrupt the database or leave influxdb3 consuming high CPU when idle.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100