sassoftware / sassoftware/python-swat

Optimizing data upload transfer size

Open
#93 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
159
Forks
65
PR merge metrics
No merged PRs in 30d

Description

Hi, in my project I usually upload from local very large tables with the following characteristic: all columns are numeric (double).

I saw that with upload_frame data is first converted to csv and then the csv is uploaded on CAS.

I was thinking about the fact that in the "plain" csv every character takes 1 byte, so for instance the number 123456.123456 uses 13 bytes: but if this number is converted to its float binary representation, it would take 8 bytes: this little "gain" of 5 bytes, repeated for every row and every column may lead to a huge gain in file transfer size for bandwidth and time.

Or perhaps the "plain" csv may be gzipped to greatly reduce the file size since inside there are only a few distinct characters (i.e. the numbers).

Any Idea/hint you can give me to optimize the data upload transfer size in a smart way?

Thank you.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the upload_frame path that converts data to CSV before uploading it to CAS. Investigate whether binary or compressed transfer is supported, then measure uploads of large all-numeric tables; done means a documented or implemented approach that reduces transfer size without breaking upload behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.