Troubleshooting Slow MIMIC-III gz Data Import into PostgreSQL Database
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 3.4k
- Forks
- 1.7k
- PR merge metrics
- No merged PRs in 30d
Description
I imported the mimic3 data into the database using postpreSQL and 7zip commands:
DROP DATABASE IF EXISTS mimic;
CREATE DATABASE mimic OWNER postgres;
\c mimic;
CREATE SCHEMA mimiciii;
set search_path to mimiciii;
\i D:/postgraduate/doctor/second/mimic3/postgres_create_tables.sql
\set ON_ERROR_STOP 1
\set mimic_data_dir 'D:/postgraduate/doctor/second/mimic3'
\i D:/postgraduate/doctor/second/mimic3/postgres_load_data_7zip.sql
After using these commands, my program stopped updating and got stuck importing the chartevent table for a long time:
mimic=# \i D:/postgraduate/doctor/second/mimic3/postgres_load_data_7zip.sql
COPY 58976
COPY 34499
COPY 7567
My machine hardware is win10, 16GB memory, no GPU, the hard disk storing the csv.gz data has 50GB free space. Is this normal? Are there ways to see the progress or diagnose if it's working correctly?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading postgres_load_data_7zip.sql and postgres_create_tables.sql, then examine the chartevent import where execution appears to stop. Check whether PostgreSQL is still processing the COPY and determine whether the behavior is normal or indicates a failure; done means documenting the diagnosis and any confirmed progress or error signals.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100