electric-sql / electric-sql/pglite

Initial Bulk Loading Big Database With Indexing

Open
#763 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
16k
Forks
442
Avg merge
20h 19m
Merged PRs (30d)
7

Description

Hi!

I am relying on the library to load some data in a PWA app which bulk loads large amount of data using the `COPY` command (with `.csv` files). I have a specific table i.e `big_table` which has roughly 120k rows and I have a `pg_trgm` index on it for fuzzy matching on the client side.
```sql
CREATE INDEX IF NOT EXISTS trgm_idx_slide_big_table
ON big_table
USING GIST ((
coalesce(remove_diacritics(content), '') || ' ' ||
coalesce(remove_diacritics(header), '')) gist_trgm_ops );
```

The current problem is that indexing alone takes around the same amount of time of bulk importing the data. The data import takes around ~10s and the Indexing takes around ~14s.

The whole database is around ~130 MB. Is there any other efficient way that I could use for faster insertion initially?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.