developmentseed / developmentseed/label-maker

Missing QA tiles create empty labels at country boundaries

Open
#106 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
472
Forks
106
PR merge metrics
No merged PRs in 30d

Description

After seeing really poor results in my ResNet50 I have manually inspected the data and labels and it appears that the background images label_maker is producing are labelled almost randomly.

Here's the config that I'm using:
```json
{
"country": "singapore",
"bounding_box": [103.5917,1.132909,104.10712,1.487815],
"zoom": 14,
"classes": [{ "name": "human construction", "filter": ["any", ["has", "man_made"], ["has", "military"], ["has", "building"], ["has", "aerialway"], ["has", "aeroway"], ["in", "landuse", "allotments", "brownfield", "cemetery", "commercial", "construction", "depot", "farmland", "farmyard", "garages", "greenhouse_horticulture", "industrial", "landfill", "military", "orchard", "plant_nursery", "port", "quarry", "railway", "recreation_ground", "religious", "residential", "retail", "village_green", "vineyard"] ] } ],
"imagery": "http://a.tiles.mapbox.com/v4/mapbox.satellite/{z}/{x}/{y}.jpg?access_token=###(removed)",
"background_ratio": 1,
"ml_type": "classification"
}
```
Essentially what I've attempted to do in the above 'classes' is to identify tiles with any sign of human activity.
At that zoom level around 300 images are packaged in to data.npz.
Once we have the data.npz file a quick inspection of some of the imagery with the following code (I used a Jupyter Notebook) shows a high proportion of the images are labelled as not in the class above despite very clearly containing man made structures.

```python
import numpy as np
import os
# load data
npz = np.load('final_singa/data.npz')
x_train = npz['x_train']
y_train = npz['y_train']
x_test = npz['x_test']
y_test = npz['y_test']

# check out labels for first 25 images:
for i in range(0,25):
print('row:',i)
fig = plt.subplots(nrows=1, ncols=1, figsize=(10, 10))
plt.imshow(x_train[i])
if y_train[i][0] == 1:
plt.title(str('No man made'))
else:
plt.title(str('Yes man made'))
plt.tight_layout
plt.show()
plt.pause(0.0001)
```

I also tried at zoom level 15 and the same issue exists.
I'm wondering a few things:
Is the class I'm using too complex to use label-maker for this purpose?
Is the OSM tiling simply unreliable?
Is there a bug in label maker?
Am I using background_ratio incorrectly?
Am I making a stupid mistake somewhere?

Any advice greatly appreciated!

Thanks
Josh

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.