influxdata / influxdata/influxdb-client-python

Performance bottleneck in FluxCsvParser when parsing large CSV payloads (10MB+)

未关闭
#691 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

bug
主要语言
Python
星标
793
派生
186
平均合并
3 小时 2 分钟
30 天内合并 PR
1

描述

### Specifications

* Client Version: 1.48.0
* InfluxDB Version: 2.7
* Platform: macos

### Code sample to reproduce problem

```python
```
import influxdb_client
client = influxdb_client.InfluxDBClient(
url="http://localhost:8086",
token="TOKEN",
org="organization",
)
query_api = client.query_api()
query = 'from(bucket: "sensors") |> range(start: 2025-04-13T14:18:11.036Z, stop: 2025-04-13T14:33:11.036Z)'
result = query_api.query(org="matna", query=query)

### Expected behavior

runtime of this query must be same as ui influx

### Actual behavior

runtime this code is not same order

### Additional info

Hi InfluxDB team,

I've encountered a significant performance bottleneck in the FluxCsvParser class within the InfluxDB Python client when working with larger datasets.

🐞 Issue Description
When querying data (~10MB in size), the network call returns results in under 20 ms, which is excellent. However, the CSV parsing step takes over 5 seconds to complete. This introduces an unacceptable latency for high-throughput or low-latency use cases.

In contrast, using the Go client for the same query and dataset, the full query—including parsing—is completed in under 200 ms. This makes the Python client around 25x slower just in the parsing stage.

📈 Performance Benchmark
Data size: ~10MB (Flux CSV)

Query response time (network): < 20 ms

Parsing time (Python client): > 5000 ms

Parsing time (Go client): < 200 ms

🔍 Root Cause
Profiling indicates that the performance degradation is centered in the FluxCsvParser implementation. The current parsing logic in Python seems to be inefficient for large responses due to overhead in string parsing, tokenization, and possibly memory management.

💡 Suggested Improvement
To address this, I suggest reviewing the implementation of FluxCsvParser—specifically around how it handles tokenization, buffering, and line-by-line parsing. Additionally, performance could be dramatically improved by offloading the CSV parsing to a C extension (e.g., using cffi, cython, or ctypes) or integrating an existing optimized parser like libcsv or simdjson.

This would help close the gap with the Go client's performance while maintaining compatibility with the current interface.
✅ Request
Could the maintainers review the FluxCsvParser code path, especially in generate function?

Is there openness to rewriting this part as a performance-critical native extension, or at least modularizing it for optional native acceleration?

贡献指南

这个仓库没有索引到贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

首先定位 FluxCsvParser 的实现及其 generate 函数,然后使用大约 10MB 的 Flux CSV 响应,将解析与网络请求分开进行性能分析。将 Python 的解析时间与报告中的 Go-client 结果进行比较,并确定分词、缓冲或行处理的开销。完成的标准是:在保留当前客户端接口的同时,测得解析性能有所改善,并记录性能比较结果。

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
performance
Issue 类型
缺陷
难度
5/5
预计耗时
一周以上
活跃度
停滞
描述清晰度
基本清楚
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。