DDMAL / DDMAL/Calvo_classifier

Multipage Calvo Trainer failed in Rodan-staging with 5 images

Open
#55 9 comments 0 reactions 0 assignees View on GitHub
PROJECT: Calvo Classifier
Dominant language
Python
Stars
1
Forks
1
PR merge metrics
No merged PRs in 30d

Description

I was trying to run the [Multipage Calvo Trainer](https://github.com/DDMAL/Calvo_classifier#rodan-jobs-definition) `(Training model for Patchwise Analysis of Music Document`) in [Rodan-staging](https://rodan-staging.simssa.ca/), with 5 images inputs and each image has 3 _rgba - layer_ inputs: `Layer 0 (background)`,` Layer 1`, `Selected Regions` that comes from the _Pixel.js_ job in another workflow (all files related are attached below). It failed with the following error:

```
Error summary: InvalidArgumentError: output dimensions must be positive [[node functional_3/up_sampling2d/resize/ResizeNearestNeighbor (defined at code/Rodan/rodan/jobs/Calvo_classifier/training_engine_sae.py:227) ]] [Op:__inference_train_function_84115] Function call stack: train_function
```
The error details are:
```
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/celery/app/trace.py", line 412, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/celery/app/trace.py", line 704, in __protected_call__
return self.run(*args, **kwargs)
File "/code/Rodan/rodan/jobs/base.py", line 771, in run
retval = self.run_my_task(inputs, settings, arg_outputs)
File "/code/Rodan/rodan/jobs/Calvo_classifier/fast_calvo_trainer.py", line 186, in run_my_task
batch_size=batch_size,
File "/code/Rodan/rodan/jobs/Calvo_classifier/training_engine_sae.py", line 227, in train_msae
epochs=epochs,
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py", line 108, in _method_wrapper
return method(self, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py", line 1098, in fit
tmp_logs = train_function(iterator)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 780, in __call__
result = self._call(*args, **kwds)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 807, in _call
return self._stateless_fn(*args, **kwds) # pylint: disable=not-callable
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 2829, in __call__
return graph_function._filtered_call(args, kwargs) # pylint: disable=protected-access
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1848, in _filtered_call
cancellation_manager=cancellation_manager)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1924, in _call_flat
ctx, args, cancellation_manager=cancellation_manager))
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 550, in call
ctx=ctx)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError: output dimensions must be positive
[[node functional_3/up_sampling2d/resize/ResizeNearestNeighbor (defined at code/Rodan/rodan/jobs/Calvo_classifier/training_engine_sae.py:227) ]] [Op:__inference_train_function_84115]

Function call stack:
train_function
```

To replicate this issue:

The workflow I used looks like:
![image](https://user-images.githubusercontent.com/59303880/127675097-f16260df-8883-48b2-aeb8-4bbd0abafba2.png)
where the input ports are `image`, `Layer 0 (background)`,` Layer 1`, `Selected Regions` (each has five), trying with Salzinnes folios 006r, 066v, 106r, 166v, A06r, which can be found in my project in Rodan-staging (shared with devs) or [here](https://drive.google.com/file/d/1L5qOKdWQJyU4aNwALya85ots2m08HJNN/view?usp=sharing).

The setting for the Calvo Trainer was:
Maximum number of samples per label: 100
Patch width: 32
Patch height: 32
Maximum number of training epochs: 5
Batch Size: 1

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the five-image workflow and inspect the failing path from fast_calvo_trainer.py:186 into training_engine_sae.py:227. Trace the input dimensions reaching the up_sampling2d operation and confirm the Calvo Trainer completes with the supplied settings without the positive-dimensions error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.