google-research / google-research/t5x

SeqIO evaluation hangs on final batch when dataset is padded

Open
#741 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3k
Forks
338
PR merge metrics
No merged PRs in 30d

Description

Hi all,

During SeqIO inference evaluation, the data pipeline is hanging for a long period of time on the final batch of data **only when the dataset is padded**. Sometimes for hours depending on the dataset size. This relates to these [lines of code](https://github.com/google-research/t5x/blob/7a10c1ff5d2ef5714c3b4d32192e5ee6b843d56f/t5x/utils.py#L1126-L1146).

This is not an issue when the dataset is not padded, i.e. when the dataset length is divisible by batch size.

I am guessing it is the `pad_ds` on this [line](https://github.com/google-research/t5x/blob/7a10c1ff5d2ef5714c3b4d32192e5ee6b843d56f/t5x/utils.py#L1144-L1145) that is very slow to process, but I am not sure why:

```python
pad_ds = ds.take(1).map(lambda i, x: (np.int64(-1), x)).repeat(
dataset_pad_amt)
```

Any idea how we could speed up the data pipeline when padding?

Contributor guide

Open the contributing guide

Research direction

Start in t5x/utils.py at lines 1126-1146 and reproduce SeqIO inference evaluation with a padded dataset, comparing it with a length divisible by the batch size. Investigate the pad_ds construction and confirm the final batch completes without the prolonged hang while preserving evaluation behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.