pabhila / pabhila/github-actions-test
[Bug]: Application crashes with "MemoryError" when processing >1GB CSV files
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
The application fails to process large CSV datasets. When attempting to upload a file larger than 1GB via the data import tool, the spinner loads for 10 seconds, and then the application crashes entirely, forcing a restart of the local server.
Steps to Reproduce
Launch the application (npm start).
Navigate to the Import Data tab.
Click Upload CSV.
Select a .csv file larger than 1GB (e.g., large_dataset.csv).
Click Process.
Expected Behavior
The file should process in chunks (streaming), or a progress bar should appear. If the file is too large, a user-friendly error message ("File too large") should be displayed instead of a hard crash.
Actual Behavior
The UI freezes, and the backend process terminates with a fatal Python error.
Screenshots/Logs
Console Output:
Plaintext
Processing file...
Traceback (most recent call last):
File "data_loader.py", line 42, in load_data
df = pd.read_csv(file_path)
pandas.errors.ParserError: Error tokenizing data. C error: MemoryError
[Process exited with code 1]
Environment
OS: Windows 11 Pro / macOS Sequoia 15.1
Browser: Chrome 120.0
Version: v2.4.1
Python Version: 3.11
Possible Solution (Optional)
It looks like we are loading the entire dataframe into memory using pandas.read_csv(). We should probably switch to chunksize=1000 to handle large files or stream the data.
Additional Context
This issue does not occur with files smaller than 500MB.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in data_loader.py at line 42 and reproduce the failure with npm start using a CSV larger than 1GB. Review how pandas.read_csv() handles the import and determine the project-supported behavior for oversized files. Done means large files no longer terminate the backend, and an appropriate user-facing outcome is verified for the import flow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100