mapbox / mapbox/mapboxgl-jupyter
Significantly faster df_to_geojson function for file output (8x)
- Dominant language
- Python
- Stars
- 681
- Forks
- 137
- PR merge metrics
- No merged PRs in 30d
Description
Hey this should probably be a pull but some of the point simplification I'd guess from the geojson package does made it to much of pain to write tests. Anyway heres a file that implements df_to_geojson in effectively the exact same way but much faster (I think)
Nothing crazy is being done here just utilizing pandas methods to create the geometry string and then to_json off the dataframe for our properties wrapped in a list compression. Anyway I figure it could be useful at least to look at.
```python
import mapboxgl.utils as old
import time
import pandas as pd
import random
import json
################################################################################################################################################################################################
# THESE FUNCTIONS EXIST ONLY TO CREATE A POINT DATA SET
CHARS = 'abcdefghijklmnopqurstuvwxyz0123456789'
SIZECHARS = len(CHARS)
def random_int():
return random.randint(0,10000)
def random_char():
return CHARS[random.randint(0,SIZECHARS-1)]
def random_string():
i = 0
string = ""
while i < 10:
string+= random_char()
i+= 1
return string
def random_float():
return random.uniform(0.0,10000.0)
# generates a number of random values
def generate_rands(number,func):
newlist = []
i = 0
while i < number:
newlist.append(func())
i+=1
return newlist
#
def random_point():
return [random.uniform(-180.0,180.0),random.uniform(-90.0,90.0)]
################################################################################################################################################################################################
def geometry(x):
return '{"type": "Point","coordinates": [%s,%s]}' % (x.LONG,x.LAT)
def df_to_geojson(df, properties=None, lat='lat', lon='lon', precision=6, filename=None):
"""Serialize a Pandas dataframe to a geojson format Python dictionary
"""
if not properties:
# if no properties are selected, use all properties in dataframe
properties = [c for c in df.columns if c not in [lon, lat]]
for prop in properties:
# Check if list of properties exists in dataframe columns
if prop not in list(df.columns):
raise ValueError(
'properties must be a valid list of column names from dataframe')
if prop in [lon, lat]:
raise ValueError(
'properties cannot be the geometry longitude or latitude column')
if filename:
with open(filename, 'w') as f:
# Write out file to line
f.write('{"type": "FeatureCollection", "features": [' +
','.join(['''{"geometry": %s, "type": "Feature", "properties": %s}''' % (geom,properties) \
for geom,properties in zip(
df[[lon,lat]].apply(geometry,axis=1).values.tolist(),
df[properties].to_json(orient='records',lines=True).splitlines()
)]) +
']}')
return {
"type": "file",
"filename": filename,
"feature_count": df.shape[0]
}
else:
features = []
df[[lon, lat] + properties].apply(lambda x: features.append(
old.row_to_geojson(x, lon, lat, precision)), axis=1)
return geojson.FeatureCollection(features)
# generating the dataframe to benchmark with
# 10k points 1 string,float, and int field respectively
number_of_rows = 10000
data = pd.DataFrame(generate_rands(number_of_rows,random_point),columns=['LONG','LAT'])
data['COL1'] = generate_rands(len(data),random_int)
data['COL2'] = generate_rands(len(data),random_string)
data['COL3'] = generate_rands(len(data),random_float)
print number_of_rows,'data created'
s = time.time()
for i in range(5):
old.df_to_geojson(data,properties=data.columns[2:].values.tolist(),lon='LONG',lat='LAT',filename='a.geojson')
e = time.time() - s
opspeed1 = (e / 5.)
print 'secs / op: %s' % opspeed1
s = time.time()
for i in range(5):
df_to_geojson(data,properties=data.columns[2:].values.tolist(),lon='LONG',lat='LAT',filename='b.geojson')
e = time.time() - s
opspeed2 = (e / 5.)
print 'secs / op: %s' % opspeed2
print "%sx faster" % (opspeed1 / opspeed2)
'''
comapre values here
with open('a.geojson','rb') as f:
data = f.read()
oldfile = json.loads(data.replace('\n',''))
with open('b.geojson','rb') as f:
data = f.read()
newfile = json.loads(data)
for oldfeat,newfeat in zip(oldfile['features'],newfile['features']):
# compare the old and the new here
pass
'''
```
### Output
```
10000 data created
secs / op: 4.89236383438
secs / op: 0.54987282753
8.89726422082x faster
```
Contributor guide
Research direction
Start with the existing df_to_geojson function and its old.row_to_geojson path, then review the provided file-output implementation. Run the 10,000-point benchmark and compare a.geojson with b.geojson; done means the output remains equivalent while file generation is faster.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100