ckan / ckan/ideas

Datastore extension for Bonobo

Open
#199 2 comments 2 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
39
Forks
1
PR merge metrics
No merged PRs in 30d

Description

[Bonobo](https://www.bonobo-project.org/) is data processing toolkit for building ETL graphs in Python.

It would be really neat if there were a Datastore extension, in a similar vein to the [opendatasoft extension](https://github.com/python-bonobo/bonobo/blob/0.2/bonobo/ext/opendatasoft.py) so that users build pipelines like

```python

#from https://github.com/python-bonobo/bonobo/blob/0.2/bonobo/examples/datasets/coffeeshops.py

from os.path import dirname, realpath, join

import bonobo
from bonobo.ext.opendatasoft import OpenDataSoftAPI

OUTPUT_FILENAME = realpath(join(dirname(__file__), 'coffeeshops.txt'))

graph = bonobo.Graph(
OpenDataSoftAPI(dataset='liste-des-cafes-a-un-euro', netloc='opendata.paris.fr'),
lambda row: '{nom_du_cafe}, {adresse}, {arrondissement} Paris, France'.format(**row),
bonobo.FileWriter(path=OUTPUT_FILENAME),
)

if __name__ == '__main__':
bonobo.run(graph)
print('Import done, read {} for results.'.format(OUTPUT_FILENAME))
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading bonobo/ext/opendatasoft.py and the referenced datasets/coffeeshops.py example to understand the existing extension and pipeline shape. Determine the intended Datastore extension entry point and usage from those references; done should be a documented extension that supports the proposed datastore-backed pipeline.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.