ckan / ckan/datapusher

Prevent DataPusher from Uploading Resources with More than X Columns

Open
#178 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
82
Forks
157
PR merge metrics
No merged PRs in 30d

Description

Hi everyone. We have an open portal featuring the Datapusher. People can create their own accounts and create their own datasets and resources. Most of things work fine except for the fact that some users have huge tabular files. Some of these files have thousands of columns. Knowinf that there are some limitations on PostgreSQL concerning the amount of bearable columns, we would like to prevent the DataPusher to try to push a resource, if it happens to be that this very resource has more than X amount of columns.

Indeed, currently, the DataPusher ust tries to send the files, no matter their size. It will of course end up being rejected due to PostgreSQL limitations, but in the meantime, it will eatup server resources (RAM, CPU, SWAP) for nothing. And some of our users have hundreds of files like this and they have automated the uploading process through API. So when they are actually sending new resources, our server is almost always dying.

Is there a way to indicate to the DataPusher not to push a resource, in case it has more than X amount of columns?

CKAN Version : 2.4.1
Datapusher Version : 0.0.6

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.