deepseek-ai / deepseek-ai/DeepSpec

Concurrent training data generation can corrupt resume order and buffer unbounded results

Open
#35 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.1k
Forks
667
PR merge metrics
No merged PRs in 30d

Description

## Problem

`generate_train_data.py --resume` uses the number of existing output/error rows to decide how many input rows to skip. That assumes output rows are written in the same order as input rows.

Concurrent generation can finish out of order. If the script writes results as futures complete, a crash can leave output rows for a non-contiguous set of input rows. On resume, the script then skips the wrong prefix, causing some samples to be regenerated and others to be skipped.

The ordered-write fix also needs bounded buffering: if an early request is slow, later completed samples should not accumulate without limit behind it.

## Impact

Large data-generation jobs can produce corrupted regenerated datasets after interruption/resume. Without bounded ordered buffering, high-concurrency runs can also retain too many completed samples in memory behind one slow earlier request.

## Fix

Fixed in PR #29 by writing generated rows in input order, validating JSONL paths, hardening JSONL loading for blank/empty files, and bounding outstanding ordered results so completed-but-not-yet-writable samples still apply backpressure.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.