influxdata / influxdata/influxdb

Support Large CSV File Imports in Influxdb3-explorer

Open
#26,622 0 comments 0 reactions 0 assignees View on GitHub
v3 v3-data-explorer
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Use case:__
> - influxdb3-explorer version 1.0.3
> - influxdb 3 core version 3.2.1

Users use influxdb3-explorer to import CSV files into InfluxDB 3 core. Real-world datasets (e.g., IoT sensor logs, monitoring data, financial records) often over 5MB or even 100MB . But the current default limits prevent successful import, blocking a core use case of the influxdb3-explorer tool.

__Proposal:__
Increase the default request body size limits in both Nginx and the Node.js backend to support large CSV file imports (e.g., up to 100MB). Make these limits configurable via environment variables or config files, and document the defaults.

__Current behaviour:__

Image

Image

Image

When importing a 5.3MB CSV file , the following actions fail:
- 【Parse Data】
- 【Convert to Line Protocol】

This is due to:
- Nginx default limit : client_max_body_size is missing in nginx.conf → defaults to 1MB .
- Node.js (NestJS) limit : The backend enforces a default express.json() limit (likely 10MB or lower), even after Nginx fix.

> Your provided nginx.conf does not include client_max_body_size, confirming the issue.

__Desired behaviour:__
1. Out-of-the-box support for CSV files up to 5-100MB without manual configuration.
2. Configurable limits via environment variables or volume file.
3. Clear documentation about limits and how to adjust them.
4. Graceful error messages if limits are exceeded

__Alternatives considered:__
1. Update nginx.conf (in image):

Image

2. Update src/main.ts (source code):
I was unable to locate the source code, but I did my utmost to find and modify the `main.js` file as follows. Fortunately, it worked!
Steps:
1. Locate and Modify the File Size Limit
Find the line "limits:{fileSize:5242880}" and change it to "limits:{fileSize:104857600}". This increases the file size limit from 5MB to 100MB.
2. Insert Express Middleware Configuration
Locate the line "i.NestFactory.create(l.AppModule,t)" and insert the following code immediately after it:
```
a.use(require("express").json({ limit: "100mb" })),a.use(require("express").urlencoded({ extended: true, limit: "100mb" })),
```
Image

By following these steps, the application should now support larger file uploads and handle requests with larger payloads. Oh, don`t forget restart the container!

Contributor guide

Open the contributing guide

Research direction

Inspect the referenced nginx.conf and the Node.js backend entry point, identified as src/main.ts, to trace the request-size limits for CSV parsing and conversion. Verify the behavior with the reported 5.3MB file and larger inputs. Done means supported limits are configurable, documented, and produce a clear error when exceeded.

Written by the indexing model from the issue text.

Assessment

Tech stack
nginx, node.js
Domain
backend, devops
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.