deepspeedai / deepspeedai/DeepSpeed
[Question] DeepSpeed + HuggingFace Trainer Displays a File After Successful Execution. How to Disable it?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
There is a file displayed in the terminal after DeepSpeed has successfully run. The output file looks like the following:
NAME
qa_finetuning.py --local_rank=0 --model_path=google/flan-t5-small --dataset_dir=qa_data --deepspeed=deepspeed_configs.json
--yaml_configs=qa_finetuning.yaml --output_dir=./tmp_model - Two-dimensional, size-mutable, potentially heterogeneous tabular
data.
SYNOPSIS
qa_finetuning.py --local_rank=0 --model_path=google/flan-t5-small --dataset_dir=qa_data --deepspeed=deepspeed_configs.json
--yaml_configs=qa_finetuning.yaml --output_dir=./tmp_model - GROUP | COMMAND | VALUE
DESCRIPTION
Data structure also contains labeled axes (rows and columns).
Arithmetic operations align on both row and column labels. Can be
thought of as a dict-like container for Series objects. The primary
pandas data structure.
GROUPS
GROUP is one of the following:
T
Two-dimensional, size-mutable, potentially heterogeneous tabular data.
at
Access a single value for a row/column label pair.
attrs
axes
columns
Immutable sequence used for indexing and alignment. The basic object storing axis labels for all pandas objects.
dtypes
One-dimensional ndarray with axis labels (including time series).
epoch
One-dimensional ndarray with axis labels (including time series).
eval_cls_f1
One-dimensional ndarray with axis labels (including time series).
eval_cls_mean_f1
One-dimensional ndarray with axis labels (including time series).
eval_cls_mean_precision
......
transpose
Transpose index and columns.
truediv
Get Floating division of dataframe and other, element-wise (binary operator `truediv`).
truncate
Truncate a Series or DataFrame before and after some index value.
tz_convert
Convert tz-aware axis to target time zone.
tz_localize
Localize tz-naive index of a Series or DataFrame to target time zone.
unstack
Pivot a level of the (necessarily hierarchical) index labels.
update
Modify in place using non-NA values from another DataFrame.
value_counts
Return a Series containing counts of unique rows in the DataFrame.
var
Return unbiased variance over requested axis.
where
Replace values where the condition is False.
xs
Return cross-section from the Series/DataFrame.
VALUES
VALUE is one of the following:
empty
ndim
size
values
(END)
When I submit the training job to Vertex AI Training, the training just stuck at this stage forever because Vertex AI Training thinks that the training has not finished. My question is there a way to avoid displaying this file after the training has finished? Or is there a way to automatically close this file so that Vertex AI knows that the training is completed?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the qa_finetuning.py command with the shown DeepSpeed and YAML configuration, then inspect why the pandas documentation text appears after execution. Check whether the process actually exits after this output; done means the command terminates cleanly and Vertex AI Training recognizes completion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, pandas, python
- Domain
- cloud, distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100