EleutherAI / EleutherAI/pile_dedupe

the minhash generation process is very slow

Open
#1 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
18
Forks
1
PR merge metrics
No merged PRs in 30d

Description

I test generate_minhashes.py with a 64 kernels server. It is very slow. Even for a 19G datasets, it requires 1 day for computing. Is it normal?

Thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

The report identifies generate_minhashes.py and a 19G dataset processed on a 64-kernel server. Start by reproducing or measuring the reported one-day runtime, then determine whether it is expected and identify what evidence is needed to define a fix; done means clarifying the expected performance or addressing a confirmed bottleneck.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.