snowflakedb / snowflakedb/snowpark-python

SNOW-682238: Long running program iterating over data with to_local_iterator crashing unexpectedly

Open
#496 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature
Dominant language
Python
Stars
341
Forks
155
Avg merge
4d 16h
Merged PRs (30d)
27

Description

  1. What version of Python are you using?
Python 3.8.9 (default, Apr  6 2022, 10:43:36)
[GCC 9.4.0]
  1. What operating system and processor architecture are you using?
Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.29
  1. What are the component versions in the environment (pip freeze)?
snowflake-snowpark-python==0.9.0
  1. What did you do?

To summarize, I have a long running program iterating over a huge table. This job performs a lot of processing on the data for every iteration. The code looks something like this:

import gc
import sys

from snowflake.snowpark import (
    DataFrame,
    Session
)

from snowflake.snowpark.row import Row

snowflake_connect_options = {
   "account": os.environ['SNOWFLAKE_ACCOUNT'],
   "user": os.environ['SNOWFLAKE_USER'],
   "password": os.environ['SNOWFLAKE_PASSWORD'],
   "warehouse": os.environ['SNOWFLAKE_WAREHOUSE'],
   "role": os.environ['SNOWFLAKE_ROLE'],
   "database": os.environ['SNOWFLAKE_DATABASE'],
   "client_session_keep_alive": True,
}

session = Session.builder.configs(snowflake_connect_options).create()

local_iterator = self.session.table(qualified_raw_table).to_local_iterator()

data_list = []

for row_data in local_iterator:
   # Perform processing operations on row_data -> processed_data
   data_list.append(processed_data)

   data_size_mb = (sys.getsizeof(data_list) + sum([sys.getsizeof(data_list[x]) for x in range(len(data_list))])) // 1000000

   if data_size_mb  > 500:
      # Write data_list contents to destination
      del data_list[:]
   gc.collect()
  1. What did you expect to see?

I expect the code to run through the local_iterator without issue, instead this happened after 6 hours (meaning it's not the 4 hour timeout messing with me):

File "/path/to/code.py", line X, in code_function
   for row_data in local_iterator:
File "/usr/local/lib/python3.8/site-packages/snowflake/snowpark/dataframe.py", line 505, in to_local_iterator
   yield from self._session._conn.execute(
File "/usr/local/lib/python3.8/site-packages/snowflake/snowpark/_internal/utils.py", line 478, in result_set_to_iter
   row = Row(*data)
TypeError: type object argument after * must be an iterable, not NoneType
  1. Can you set logging to DEBUG and collect the logs?

Unfortunately it's hard for me to do this as the job takes a long time to run and generates a lot of other logs.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at snowpark/dataframe.py's to_local_iterator entry point and follow the result_set_to_iter path in snowpark/_internal/utils.py, where the traceback shows Row receives None instead of iterable data. Reproduce or isolate the long-running iteration failure and determine what happens after several hours. Done means the iterator completes without this TypeError under the reported usage.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.