developmentseed / developmentseed/label-maker
Missing QA tiles create empty labels at country boundaries
- Dominant language
- Python
- Stars
- 472
- Forks
- 106
- PR merge metrics
- No merged PRs in 30d
Description
After seeing really poor results in my ResNet50 I have manually inspected the data and labels and it appears that the background images label_maker is producing are labelled almost randomly.
Here's the config that I'm using:
```json
{
"country": "singapore",
"bounding_box": [103.5917,1.132909,104.10712,1.487815],
"zoom": 14,
"classes": [{ "name": "human construction", "filter": ["any", ["has", "man_made"], ["has", "military"], ["has", "building"], ["has", "aerialway"], ["has", "aeroway"], ["in", "landuse", "allotments", "brownfield", "cemetery", "commercial", "construction", "depot", "farmland", "farmyard", "garages", "greenhouse_horticulture", "industrial", "landfill", "military", "orchard", "plant_nursery", "port", "quarry", "railway", "recreation_ground", "religious", "residential", "retail", "village_green", "vineyard"] ] } ],
"imagery": "http://a.tiles.mapbox.com/v4/mapbox.satellite/{z}/{x}/{y}.jpg?access_token=###(removed)",
"background_ratio": 1,
"ml_type": "classification"
}
```
Essentially what I've attempted to do in the above 'classes' is to identify tiles with any sign of human activity.
At that zoom level around 300 images are packaged in to data.npz.
Once we have the data.npz file a quick inspection of some of the imagery with the following code (I used a Jupyter Notebook) shows a high proportion of the images are labelled as not in the class above despite very clearly containing man made structures.
```python
import numpy as np
import os
# load data
npz = np.load('final_singa/data.npz')
x_train = npz['x_train']
y_train = npz['y_train']
x_test = npz['x_test']
y_test = npz['y_test']
# check out labels for first 25 images:
for i in range(0,25):
print('row:',i)
fig = plt.subplots(nrows=1, ncols=1, figsize=(10, 10))
plt.imshow(x_train[i])
if y_train[i][0] == 1:
plt.title(str('No man made'))
else:
plt.title(str('Yes man made'))
plt.tight_layout
plt.show()
plt.pause(0.0001)
```
I also tried at zoom level 15 and the same issue exists.
I'm wondering a few things:
Is the class I'm using too complex to use label-maker for this purpose?
Is the OSM tiling simply unreliable?
Is there a bug in label maker?
Am I using background_ratio incorrectly?
Am I making a stupid mistake somewhere?
Any advice greatly appreciated!
Thanks
Josh
Contributor guide
Assessment
This issue has not been assessed yet.