tensorflow / tensorflow/model-optimization

prune_low_magnitude fail when wrapped with GRU layer, but success with conv2d and dense

Open
#944 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
1.6k
Forks
349
Avg merge
3d 2h
Merged PRs (30d)
1

Description

Prior to filing: check that this should be a bug instead of a feature request. Everything supported, including the compatible versions of TensorFlow, is listed in the overview page of each technique. For example, the overview page of quantization-aware training is here. An issue for anything not supported should be a feature request.

Describe the bug
A clear and concise description of what the bug is.

System information

TensorFlow version (installed from source or binary): 2.7.0

TensorFlow Model Optimization version (installed from source or binary): 0.7.1

Python version: 3.8

Describe the expected behavior

Describe the current behavior
Hi, i can use the prune_low_magnitude for Dense and Conv2D. And When use the prune_low_magnitude, it failed when wrapped with GRU.

keras.layers.GRU(units=numUnits, 
unroll=False,
return_sequences=True, 
recurrent_activation='sigmoid',
return_state=False)

Also, when i remove the ModelCheckpoint callback, it can run with GRU, but i cannot save the model.

checkpoint_callback = ModelCheckpoint(os.path.join(exp_dir, 'model_{epoch:02d}.h5'),
                                       monitor='val_loss',
                                       save_best_only=False,
                                       save_weights_only=False,
                                       mode='auto',
                                       save_freq='epoch')

Code to reproduce the issue
Provide a reproducible code that is the bare minimum necessary to generate the
problem.

Screenshots
If applicable, add screenshots to help explain your problem.

2022-03-15 15:41:00.249444: I tensorflow/stream_executor/cuda/cuda_dnn.cc:366] Loaded cuDNN version 8101
2022-03-15 15:41:00.626781: I tensorflow/core/platform/default/subprocess.cc:304] Start cannot spawn child process: No such file or directory
10/10 [==============================] - ETA: 0s - loss: 867.6863 - ola_layer_loss: 6.6291 - mask_layer_loss: 16.1240 - vad_output_loss: 1.0971/home/test/dev/anaconda3/envs/tf2.7_py3.8/lib/python3.8/site-packages/keras/engine/functional.py:1410: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.
  layer_config = serialize_layer_fn(layer)
Traceback (most recent call last):
  File "dpcrn/bin/train.py", line 97, in <module>
    main()
  File "dpcrn/bin/train.py", line 87, in main
    excuctor.train(loss_wrapper=loss_wrapper, 
  File "/home/test/code/speech_enhance/DPCRN_DNS3/dpcrn/utils/executor.py", line 136, in train
    self.model.fit_generator(data_generator.generator(batch_size=batch_size, validation=False), 
  File "/home/test/dev/anaconda3/envs/tf2.7_py3.8/lib/python3.8/site-packages/keras/engine/training.py", line 2016, in fit_generator
    return self.fit(
  File "/home/test/dev/anaconda3/envs/tf2.7_py3.8/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/test/dev/anaconda3/envs/tf2.7_py3.8/lib/python3.8/site-packages/h5py/_hl/group.py", line 149, in create_dataset
    dsid = dataset.make_new_dset(group, shape, dtype, data, name, **kwds)
  File "/home/test/dev/anaconda3/envs/tf2.7_py3.8/lib/python3.8/site-packages/h5py/_hl/dataset.py", line 142, in make_new_dset
    dset_id = h5d.create(parent.id, name, tid, sid, dcpl=dcpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5d.pyx", line 87, in h5py.h5d.create
ValueError: Unable to create dataset (name already exists)
2022-03-15 15:41:04.771988: W tensorflow/core/kernels/data/generator_dataset_op.cc:107] Error occurred when finalizing GeneratorDataset iterator: FAILED_PRECONDITION: Python interpreter state is not initialized. The process may be terminated.
         [[{{node PyFunc}}]]

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the GRU wrapped by prune_low_magnitude and the ModelCheckpoint configuration in the supplied example, then trace the save call through dpcrn/bin/train.py and dpcrn/utils/executor.py. Reproduce the HDF5 “name already exists” failure and determine whether it is caused by GRU pruning or model checkpoint serialization; done means the supported GRU case can train and save without that error.

Written by the indexing model from the issue text.

Assessment

Tech stack
keras, python, tensorflow
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.