deepspeedai / deepspeedai/DeepSpeed

[Question] DeepSpeed + HuggingFace Trainer Displays a File After Successful Execution. How to Disable it?

Open
#4,092 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 15h
Merged PRs (30d)
112

Description

There is a file displayed in the terminal after DeepSpeed has successfully run. The output file looks like the following:

NAME
    qa_finetuning.py --local_rank=0 --model_path=google/flan-t5-small --dataset_dir=qa_data --deepspeed=deepspeed_configs.json
 --yaml_configs=qa_finetuning.yaml --output_dir=./tmp_model - Two-dimensional, size-mutable, potentially heterogeneous tabular
 data.

SYNOPSIS
    qa_finetuning.py --local_rank=0 --model_path=google/flan-t5-small --dataset_dir=qa_data --deepspeed=deepspeed_configs.json
 --yaml_configs=qa_finetuning.yaml --output_dir=./tmp_model - GROUP | COMMAND | VALUE

DESCRIPTION
    Data structure also contains labeled axes (rows and columns).
    Arithmetic operations align on both row and column labels. Can be
    thought of as a dict-like container for Series objects. The primary
    pandas data structure.

GROUPS
    GROUP is one of the following:

     T
       Two-dimensional, size-mutable, potentially heterogeneous tabular data.

     at
       Access a single value for a row/column label pair.

     attrs

     axes

     columns
       Immutable sequence used for indexing and alignment. The basic object storing axis labels for all pandas objects.

     dtypes
       One-dimensional ndarray with axis labels (including time series).

     epoch
       One-dimensional ndarray with axis labels (including time series).

     eval_cls_f1
       One-dimensional ndarray with axis labels (including time series).

     eval_cls_mean_f1
       One-dimensional ndarray with axis labels (including time series).

     eval_cls_mean_precision

......
     transpose
       Transpose index and columns.

     truediv
       Get Floating division of dataframe and other, element-wise (binary operator `truediv`).

     truncate
       Truncate a Series or DataFrame before and after some index value.

     tz_convert
       Convert tz-aware axis to target time zone.

     tz_localize
       Localize tz-naive index of a Series or DataFrame to target time zone.

     unstack
       Pivot a level of the (necessarily hierarchical) index labels.

     update
       Modify in place using non-NA values from another DataFrame.

     value_counts
       Return a Series containing counts of unique rows in the DataFrame.

     var
       Return unbiased variance over requested axis.

     where
       Replace values where the condition is False.

     xs
       Return cross-section from the Series/DataFrame.

VALUES
    VALUE is one of the following:

     empty

     ndim

     size

     values

(END)

When I submit the training job to Vertex AI Training, the training just stuck at this stage forever because Vertex AI Training thinks that the training has not finished. My question is there a way to avoid displaying this file after the training has finished? Or is there a way to automatically close this file so that Vertex AI knows that the training is completed?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the qa_finetuning.py command with the shown DeepSpeed and YAML configuration, then inspect why the pandas documentation text appears after execution. Check whether the process actually exits after this output; done means the command terminates cleanly and Vertex AI Training recognizes completion.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, pandas, python
Domain
cloud, distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.