Augmented data is stopping weight calculation and training.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 385
- PR merge metrics
- No merged PRs in 30d
Description
@daniel-j-h Hello! I am working with Sentinel-2 data and successfully trained with 0.56 accuracy. Now I want to increase training data so I am trying data augmentation (rotation, zooming). I rotated and zoomed current data (I did the same for labels) with Keras library (ImageDataGenerator). New data has appropriate z/x/y names (continued from the last number of x).
While calculating weights when it comes to new data I am getting:
```
Traceback (most recent call last):
File "weights.py", line 69, in
main(args.dataset)
File "weights.py", line 47, in main
counts += np.bincount(image.ravel(), minlength=num_classes)
ValueError: operands could not be broadcast together with shapes (2,) (142,) (2,)
```
**What will be the reason? How should I solve it?**
Furthermore, I noticed that original data represents binary data (print (image)), while new data has values up to 255. What is a problem?
When starting the training without calculating weights I am getting:
```
/pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo, TensorInfo, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [41,0,0], thread: [98,0,0] Assertion `indexValue >= 0 && indexValue < tensor.sizes[dim]` failed.
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/src/app/robosat/tools/__main__.py", line 58, in
args.func(args)
File "/usr/src/app/robosat/tools/train.py", line 129, in main
train_hist = train(train_loader, num_classes, device, net, optimizer, criterion)
File "/usr/src/app/robosat/tools/train.py", line 185, in train
loss = criterion(outputs, masks)
File "/opt/venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(*input, **kwargs)
File "/usr/src/app/robosat/losses.py", line 106, in forward
errors_sorted, indices = torch.sort(max_margin_errors, descending=True)
RuntimeError: merge_sort: failed to synchronize: device-side assert triggered
```
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing the original and augmented mask values and shapes, then read weights.py around line 47 and the training path in robosat/tools/train.py and robosat/losses.py. Reproduce the weight calculation and training failure with the augmented dataset. Done means the incompatible mask data or processing step is identified and the affected workflow completes without the reported errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- keras, python, pytorch
- Domain
- computer-vision, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100