MIT-LCP / MIT-LCP/mimic-code

Troubleshooting Slow MIMIC-III gz Data Import into PostgreSQL Database

Open
#1,609 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
3.4k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

I imported the mimic3 data into the database using postpreSQL and 7zip commands:

DROP DATABASE IF EXISTS mimic;

CREATE DATABASE mimic OWNER postgres;
\c mimic;
CREATE SCHEMA mimiciii;
set search_path to mimiciii;
\i D:/postgraduate/doctor/second/mimic3/postgres_create_tables.sql
\set ON_ERROR_STOP 1
\set mimic_data_dir 'D:/postgraduate/doctor/second/mimic3'
\i D:/postgraduate/doctor/second/mimic3/postgres_load_data_7zip.sql

After using these commands, my program stopped updating and got stuck importing the chartevent table for a long time:

mimic=# \i D:/postgraduate/doctor/second/mimic3/postgres_load_data_7zip.sql
COPY 58976
COPY 34499
COPY 7567

My machine hardware is win10, 16GB memory, no GPU, the hard disk storing the csv.gz data has 50GB free space. Is this normal? Are there ways to see the progress or diagnose if it's working correctly?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading postgres_load_data_7zip.sql and postgres_create_tables.sql, then examine the chartevent import where execution appears to stop. Check whether PostgreSQL is still processing the COPY and determine whether the behavior is normal or indicates a failure; done means documenting the diagnosis and any confirmed progress or error signals.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.