tensorflow / tensorflow/datasets

C4/en dataset download stopped

Open
#3,094 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

help
Dominant language
Python
Stars
4.6k
Forks
1.6k
Avg merge
3h 54m
Merged PRs (30d)
1

Description

What I need help with / What I was wondering
I am trying to download the clean version of C4/en dataset. The dataset should be 800 GB and the space I have is 2 TB. Unfortunately after some times, the training "stopped". I am disconnected from the VM and can't access it whithout rebooting it.
I was wondering what are the requirement (hardware) to download the clean version of c4? How long does it take to download it ?

What I've tried so far

import tensorflow_datasets as tfds

# Construct a tf.data.Dataset
ds = tfds.load('c4/en', split='train')

It would be nice if...
Could we have done anything to make things better (documentation, etc.)?

Environment information
(if applicable)

  • Operating System: ubuntu 18.04
  • Python version: 3.7
  • tensorflow-datasets/tfds-nightly version:
  • tensorflow/tensorflow-gpu/tf-nightly/tf-nightly-gpu version: 2.4.1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the shown tfds.load('c4/en', split='train') download with the stated Ubuntu 18.04, Python 3.7, and TensorFlow 2.4.1 environment, while recording the available hardware and failure point. Determine whether the stopped download is a reproducible defect or missing guidance; done means a confirmed fix or documentation describing requirements and expected download behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
data, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.