Prevent DataPusher from Uploading Resources with More than X Columns
- Dominant language
- Python
- Stars
- 82
- Forks
- 157
- PR merge metrics
- No merged PRs in 30d
Description
Hi everyone. We have an open portal featuring the Datapusher. People can create their own accounts and create their own datasets and resources. Most of things work fine except for the fact that some users have huge tabular files. Some of these files have thousands of columns. Knowinf that there are some limitations on PostgreSQL concerning the amount of bearable columns, we would like to prevent the DataPusher to try to push a resource, if it happens to be that this very resource has more than X amount of columns.
Indeed, currently, the DataPusher ust tries to send the files, no matter their size. It will of course end up being rejected due to PostgreSQL limitations, but in the meantime, it will eatup server resources (RAM, CPU, SWAP) for nothing. And some of our users have hundreds of files like this and they have automated the uploading process through API. So when they are actually sending new resources, our server is almost always dying.
Is there a way to indicate to the DataPusher not to push a resource, in case it has more than X amount of columns?
CKAN Version : 2.4.1
Datapusher Version : 0.0.6
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.