influxdata / influxdata/influxdb-comparisons

Bench-Marker tool is not able to read queries from a file having 1M queries

Open
#16 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
313
Forks
110
PR merge metrics
No merged PRs in 30d

Description

Use case : "Identifying a data base which supports 1.08 million events per hour"

Steps I am following :

1) I am populating data in InfluxDB by using Java API. ( Not using `bulk_data_gen` and bulk_data_load tools ) .
2) In my Data model, I have one measurement with 1 million rows. All tags and keys are String field.
3) now I am using "bulk_query_gen " tool to generate queries and write it into a file.
I have modified the tool to create queries like "select from measurement where time < X and Time < y"

**Bulk_gen_query tool**

` ./bulk_query_gen --debug=0 --seed=321 --format=influx-http --timestamp-end="2016-01-18T14:44:00Z" -queries=1000000 --query-type="1-host-1-hr" --db="benchmark" --debug 1 | gzip > EventQueries1M.gz`

** Query_Benchmarker_tool**

bash-3.2$
`cat EventQueries1M.gz | gunzip | ./query_benchmarker_influxdb --url=http://localhost:8086 --print-interval=0 –limit=1000 --workers=2 --print-responses=true -- db="benchmark" --debug 1`

`**2017/01/27 21:45:51 Error during request: dialing to the given TCP address timed out**`

( If the query size is > 10k, I am running into this issue )

Please note, I could reproduce this issue without making any changes to the benchmarking tool.

Contributor guide

No contributing guide indexed for this repository

Research direction

The report names the bulk_query_gen and query_benchmarker_influxdb entry points. First reproduce the pipeline with EventQueries1M.gz, gunzip, and more than 10k queries, then inspect where query_benchmarker_influxdb handles input and request timeouts. Done means the benchmarker can process the large query file without the reported TCP timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, java
Domain
cli, databases, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.