developmentseed / developmentseed/planet-stream

Changeset metadata intermittently fails to save to redis

Open
#18 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
30
Forks
8
PR merge metrics
No merged PRs in 30d

Description

I'm seeing a large number of "not found" messages for metadata that is missing from redis so it never gets joined and sent out. I've confirmed that planet stream is loading the changeset metadata file from OSM, but occasionally it fails to store an entire file's worth of changesets in redis.

I'm going to turn on DEBUG=\* and see if I can get and details from the redis client logs.

I'm thinking it might happen if overpass also comes back at the same time. Here are the logs around a file that didn't save:

```
Fri Jun 10 2016 21:39:16 GMT+0000 (UTC)] INFO Polling OSM planet
[Fri Jun 10 2016 21:39:19 GMT+0000 (UTC)] INFO Retrieving http://planet.osm.org/replication/changesets/001/880/581.osm.gz from planet
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] INFO Polling Overpass
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] INFO Matching 24 members
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]7085924
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]7166210
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]13897298
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]21938396
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]22146408
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39591061
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39938357
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39941586
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39941739
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942584
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942654
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942703
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942704
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942705
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942706
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942707
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942708
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942709
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942710
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942711
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942712
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942713
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942714
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG [to match]39942715
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] INFO Polling OSM planet
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG 7085924 not found
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG 7166210 not found
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG 13897298 not found
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG 21938396 not found
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG 22146408 not found
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG 39591061 not found
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG 39938357 not found
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG 39941586 not found
[Fri Jun 10 2016 21:39:46 GMT+0000 (UTC)] DEBUG 39941739 not found
```

Here is an example of all the not found errors I'm seeing in AWS.

![image](https://cloud.githubusercontent.com/assets/2431570/15980513/35a159ce-2f3a-11e6-96d5-2b91d23a943f.png)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.