influxdata / influxdata/influxdb
Support Large CSV File Imports in Influxdb3-explorer
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Use case:__
> - influxdb3-explorer version 1.0.3
> - influxdb 3 core version 3.2.1
Users use influxdb3-explorer to import CSV files into InfluxDB 3 core. Real-world datasets (e.g., IoT sensor logs, monitoring data, financial records) often over 5MB or even 100MB . But the current default limits prevent successful import, blocking a core use case of the influxdb3-explorer tool.
__Proposal:__
Increase the default request body size limits in both Nginx and the Node.js backend to support large CSV file imports (e.g., up to 100MB). Make these limits configurable via environment variables or config files, and document the defaults.
__Current behaviour:__
When importing a 5.3MB CSV file , the following actions fail:
- 【Parse Data】
- 【Convert to Line Protocol】
This is due to:
- Nginx default limit : client_max_body_size is missing in nginx.conf → defaults to 1MB .
- Node.js (NestJS) limit : The backend enforces a default express.json() limit (likely 10MB or lower), even after Nginx fix.
> Your provided nginx.conf does not include client_max_body_size, confirming the issue.
__Desired behaviour:__
1. Out-of-the-box support for CSV files up to 5-100MB without manual configuration.
2. Configurable limits via environment variables or volume file.
3. Clear documentation about limits and how to adjust them.
4. Graceful error messages if limits are exceeded
__Alternatives considered:__
1. Update nginx.conf (in image):
2. Update src/main.ts (source code):
I was unable to locate the source code, but I did my utmost to find and modify the `main.js` file as follows. Fortunately, it worked!
Steps:
1. Locate and Modify the File Size Limit
Find the line "limits:{fileSize:5242880}" and change it to "limits:{fileSize:104857600}". This increases the file size limit from 5MB to 100MB.
2. Insert Express Middleware Configuration
Locate the line "i.NestFactory.create(l.AppModule,t)" and insert the following code immediately after it:
```
a.use(require("express").json({ limit: "100mb" })),a.use(require("express").urlencoded({ extended: true, limit: "100mb" })),
```
By following these steps, the application should now support larger file uploads and handle requests with larger payloads. Oh, don`t forget restart the container!
Contributor guide
Research direction
Inspect the referenced nginx.conf and the Node.js backend entry point, identified as src/main.ts, to trace the request-size limits for CSV parsing and conversion. Verify the behavior with the reported 5.3MB file and larger inputs. Done means supported limits are configurable, documented, and produce a clear error when exceeded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- nginx, node.js
- Domain
- backend, devops
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100