mapbox / mapbox/mapboxgl-jupyter

Significantly faster df_to_geojson function for file output (8x)

Open
#96 5 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
Python
Stars
681
Forks
137
PR merge metrics
No merged PRs in 30d

Description

Hey this should probably be a pull but some of the point simplification I'd guess from the geojson package does made it to much of pain to write tests. Anyway heres a file that implements df_to_geojson in effectively the exact same way but much faster (I think)

Nothing crazy is being done here just utilizing pandas methods to create the geometry string and then to_json off the dataframe for our properties wrapped in a list compression. Anyway I figure it could be useful at least to look at.

```python
import mapboxgl.utils as old
import time
import pandas as pd
import random
import json

################################################################################################################################################################################################
# THESE FUNCTIONS EXIST ONLY TO CREATE A POINT DATA SET

CHARS = 'abcdefghijklmnopqurstuvwxyz0123456789'
SIZECHARS = len(CHARS)

def random_int():
return random.randint(0,10000)

def random_char():
return CHARS[random.randint(0,SIZECHARS-1)]

def random_string():
i = 0
string = ""
while i < 10:
string+= random_char()
i+= 1
return string

def random_float():
return random.uniform(0.0,10000.0)

# generates a number of random values
def generate_rands(number,func):
newlist = []
i = 0
while i < number:
newlist.append(func())
i+=1
return newlist

#
def random_point():
return [random.uniform(-180.0,180.0),random.uniform(-90.0,90.0)]

################################################################################################################################################################################################

def geometry(x):
return '{"type": "Point","coordinates": [%s,%s]}' % (x.LONG,x.LAT)

def df_to_geojson(df, properties=None, lat='lat', lon='lon', precision=6, filename=None):
"""Serialize a Pandas dataframe to a geojson format Python dictionary
"""

if not properties:
# if no properties are selected, use all properties in dataframe
properties = [c for c in df.columns if c not in [lon, lat]]

for prop in properties:
# Check if list of properties exists in dataframe columns
if prop not in list(df.columns):
raise ValueError(
'properties must be a valid list of column names from dataframe')
if prop in [lon, lat]:
raise ValueError(
'properties cannot be the geometry longitude or latitude column')

if filename:
with open(filename, 'w') as f:
# Write out file to line
f.write('{"type": "FeatureCollection", "features": [' +
','.join(['''{"geometry": %s, "type": "Feature", "properties": %s}''' % (geom,properties) \
for geom,properties in zip(
df[[lon,lat]].apply(geometry,axis=1).values.tolist(),
df[properties].to_json(orient='records',lines=True).splitlines()
)]) +
']}')

return {
"type": "file",
"filename": filename,
"feature_count": df.shape[0]
}
else:
features = []
df[[lon, lat] + properties].apply(lambda x: features.append(
old.row_to_geojson(x, lon, lat, precision)), axis=1)
return geojson.FeatureCollection(features)

# generating the dataframe to benchmark with
# 10k points 1 string,float, and int field respectively
number_of_rows = 10000

data = pd.DataFrame(generate_rands(number_of_rows,random_point),columns=['LONG','LAT'])
data['COL1'] = generate_rands(len(data),random_int)
data['COL2'] = generate_rands(len(data),random_string)
data['COL3'] = generate_rands(len(data),random_float)
print number_of_rows,'data created'

s = time.time()
for i in range(5):
old.df_to_geojson(data,properties=data.columns[2:].values.tolist(),lon='LONG',lat='LAT',filename='a.geojson')
e = time.time() - s
opspeed1 = (e / 5.)
print 'secs / op: %s' % opspeed1

s = time.time()
for i in range(5):
df_to_geojson(data,properties=data.columns[2:].values.tolist(),lon='LONG',lat='LAT',filename='b.geojson')
e = time.time() - s
opspeed2 = (e / 5.)
print 'secs / op: %s' % opspeed2

print "%sx faster" % (opspeed1 / opspeed2)

'''
comapre values here
with open('a.geojson','rb') as f:
data = f.read()
oldfile = json.loads(data.replace('\n',''))

with open('b.geojson','rb') as f:
data = f.read()
newfile = json.loads(data)

for oldfeat,newfeat in zip(oldfile['features'],newfile['features']):
# compare the old and the new here
pass
'''
```

### Output
```
10000 data created
secs / op: 4.89236383438
secs / op: 0.54987282753
8.89726422082x faster
```

Contributor guide

Open the contributing guide

Research direction

Start with the existing df_to_geojson function and its old.row_to_geojson path, then review the provided file-output implementation. Run the 10,000-point benchmark and compare a.geojson with b.geojson; done means the output remains equivalent while file generation is faster.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.