snowflakedb / snowflakedb/snowpark-python

SNOW-893210: NotImplementedError when using SnowflakeFile to read from External Staging in UDF

Open
#1,006 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug triaged
Dominant language
Python
Stars
341
Forks
155
Avg merge
4d 16h
Merged PRs (30d)
27

Description

Please answer these questions before submitting your issue. Thanks!

  1. What version of Python are you using?

    3.8

  2. What operating system and processor architecture are you using?

"Linux-5.4.129-200.el7.x86_64-x86_64-with-glibc2.17"

  1. What are the component versions in the environment (pip freeze)?

  2. What did you do?

CREATE OR REPLACE FUNCTION "LOAD_JSON"(file_path string)
RETURNS variant
LANGUAGE PYTHON
RUNTIME_VERSION = '3.8'
PACKAGES = ('snowflake-snowpark-python')
HANDLER = 'process'
AS
$$
import platform
from snowflake.snowpark.files import SnowflakeFile
import json

def process(file_path):
with SnowflakeFile.open(file_path, 'r') as f:
return json.load(f)
$$;

select "LOAD_JSON"(BUILD_SCOPED_FILE_URL(<<external_blob>>, <<file_path>>));

  1. What did you expect to see?

Expecting to JSON returned in UDF, but received the following error when calling the UDF:

Traceback (most recent call last):
File "_udf_code.py", line 8, in process
File "/usr/lib/python_udf/fca69f3e27f8e37a6a12047c6ce59553c3e8fbbd8157e24384ef483acdf4ac81/lib/python3.8/json/init.py", line 293, in load
return loads(fp.read(),
File "/usr/lib/python_udf/fca69f3e27f8e37a6a12047c6ce59553c3e8fbbd8157e24384ef483acdf4ac81/lib/python3.8/site-packages/snowflake/snowpark/files.py", line 132, in read
raise NotImplementedError(_DEFER_IMPLEMENTATION_ERR_MSG)
NotImplementedError: SnowflakeFile currently only works in UDF and Stored Procedures. It doesn't work locally yet.
in function LOAD_JSON with handler process

It was working until this weekend.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the shown LOAD_JSON UDF with SnowflakeFile.open and inspect snowflake.snowpark.files.SnowflakeFile.read, which raises the reported error. Done means reading a JSON file from external staging in a UDF returns the expected value instead of NotImplementedError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.