ckan / ckan/ideas

Replace the DataProxy!

Open
#217 16 comments 9 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
39
Forks
1
PR merge metrics
No merged PRs in 30d

Description

The recline_view is able to generate previews for rectangular data. When using the datastore, I gather this works well (I don't use the datastore, so ...).

When not using the datastore, recline talks to the dataproxy which is hosted on Google App Engine, and provides a JSON view of the XLS or CSV file being previewed.

The DataProxy has several big flaws:

1. It needs the URL to the data file to be publicly accessible. Not always the case.
2. It frequently breaks.
3. It doesn't cache anything.
4. It frequently breaks.

It would be nice for there to be another dataproxy that:

* Could be hosted locally on the CKAN server
* Can access files stored locally (but not in the datastore) as well as remote URLs
* Caches the generated JSON for a period of time
* Gave the sysadmin more control over its behaviour
* Was trivially deployable
* Could save whoever it paying the GAE bill some money

Contributor guide

No contributing guide indexed for this repository

Research direction

Review the recline_view, datastore, and existing DataProxy flow first, including its Google App Engine deployment. Define how a locally hosted replacement would access local and remote files, cache generated JSON, expose sysadmin controls, and deploy on the CKAN server; done means these requirements are addressed without relying on publicly accessible URLs.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud
Domain
backend, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.