mapbox / mapbox/robosat

Augmented data is stopping weight calculation and training.

Open
#224 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.1k
Forks
385
PR merge metrics
No merged PRs in 30d

Description

@daniel-j-h Hello! I am working with Sentinel-2 data and successfully trained with 0.56 accuracy. Now I want to increase training data so I am trying data augmentation (rotation, zooming). I rotated and zoomed current data (I did the same for labels) with Keras library (ImageDataGenerator). New data has appropriate z/x/y names (continued from the last number of x).
While calculating weights when it comes to new data I am getting:
```
Traceback (most recent call last):
File "weights.py", line 69, in
main(args.dataset)
File "weights.py", line 47, in main
counts += np.bincount(image.ravel(), minlength=num_classes)
ValueError: operands could not be broadcast together with shapes (2,) (142,) (2,)
```
**What will be the reason? How should I solve it?**
Furthermore, I noticed that original data represents binary data (print (image)), while new data has values up to 255. What is a problem?

When starting the training without calculating weights I am getting:

```
/pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo, TensorInfo, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [41,0,0], thread: [98,0,0] Assertion `indexValue >= 0 && indexValue < tensor.sizes[dim]` failed.

Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/src/app/robosat/tools/__main__.py", line 58, in
args.func(args)
File "/usr/src/app/robosat/tools/train.py", line 129, in main
train_hist = train(train_loader, num_classes, device, net, optimizer, criterion)
File "/usr/src/app/robosat/tools/train.py", line 185, in train
loss = criterion(outputs, masks)
File "/opt/venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(*input, **kwargs)
File "/usr/src/app/robosat/losses.py", line 106, in forward
errors_sorted, indices = torch.sort(max_margin_errors, descending=True)
RuntimeError: merge_sort: failed to synchronize: device-side assert triggered

```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing the original and augmented mask values and shapes, then read weights.py around line 47 and the training path in robosat/tools/train.py and robosat/losses.py. Reproduce the weight calculation and training failure with the augmented dataset. Done means the incompatible mask data or processing step is identified and the affected workflow completes without the reported errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
keras, python, pytorch
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.