snowflakedb / snowflakedb/snowpark-python
SNOW-682238: Long running program iterating over data with to_local_iterator crashing unexpectedly
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 341
- Forks
- 155
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 27
Description
- What version of Python are you using?
Python 3.8.9 (default, Apr 6 2022, 10:43:36)
[GCC 9.4.0]
- What operating system and processor architecture are you using?
Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.29
- What are the component versions in the environment (
pip freeze)?
snowflake-snowpark-python==0.9.0
- What did you do?
To summarize, I have a long running program iterating over a huge table. This job performs a lot of processing on the data for every iteration. The code looks something like this:
import gc
import sys
from snowflake.snowpark import (
DataFrame,
Session
)
from snowflake.snowpark.row import Row
snowflake_connect_options = {
"account": os.environ['SNOWFLAKE_ACCOUNT'],
"user": os.environ['SNOWFLAKE_USER'],
"password": os.environ['SNOWFLAKE_PASSWORD'],
"warehouse": os.environ['SNOWFLAKE_WAREHOUSE'],
"role": os.environ['SNOWFLAKE_ROLE'],
"database": os.environ['SNOWFLAKE_DATABASE'],
"client_session_keep_alive": True,
}
session = Session.builder.configs(snowflake_connect_options).create()
local_iterator = self.session.table(qualified_raw_table).to_local_iterator()
data_list = []
for row_data in local_iterator:
# Perform processing operations on row_data -> processed_data
data_list.append(processed_data)
data_size_mb = (sys.getsizeof(data_list) + sum([sys.getsizeof(data_list[x]) for x in range(len(data_list))])) // 1000000
if data_size_mb > 500:
# Write data_list contents to destination
del data_list[:]
gc.collect()
- What did you expect to see?
I expect the code to run through the local_iterator without issue, instead this happened after 6 hours (meaning it's not the 4 hour timeout messing with me):
File "/path/to/code.py", line X, in code_function
for row_data in local_iterator:
File "/usr/local/lib/python3.8/site-packages/snowflake/snowpark/dataframe.py", line 505, in to_local_iterator
yield from self._session._conn.execute(
File "/usr/local/lib/python3.8/site-packages/snowflake/snowpark/_internal/utils.py", line 478, in result_set_to_iter
row = Row(*data)
TypeError: type object argument after * must be an iterable, not NoneType
- Can you set logging to DEBUG and collect the logs?
Unfortunately it's hard for me to do this as the job takes a long time to run and generates a lot of other logs.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at snowpark/dataframe.py's to_local_iterator entry point and follow the result_set_to_iter path in snowpark/_internal/utils.py, where the traceback shows Row receives None instead of iterable data. Reproduce or isolate the long-running iteration failure and determine what happens after several hours. Done means the iterator completes without this TypeError under the reported usage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100