sassoftware / sassoftware/python-swat
moving data from cas to data frame (pandas) takes forever . i have binary connection
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 159
- Forks
- 65
- PR merge metrics
- No merged PRs in 30d
Description
I have dataset (2,3 gb) in cas and want to bring to data frame, so I can pandas . this takes more than 30 minutes. we have a large sas viya environment (6 TB cas). i am using a binary connection, so It should work with large(r) data. this is not big data.
command bring data from cas to a data frame.
df= indata.to_frame()
why does this takes forever.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the indata.to_frame() path in the Python SWAT package and review how CAS results are transferred into pandas. Reproduce the report with a comparable dataset and binary connection, then identify whether the delay is in result retrieval or DataFrame conversion. Done means the cause and a supported fix or workaround are documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100