snowflakedb / snowflakedb/snowpark-python
SNOW-997694: Allow DataFrame object to be used as a subquery in filter operations
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 341
- Forks
- 155
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 27
Description
What is the current behavior?
Rows must be collected if they are to be used in filter operations.
This errors out:
df.filter(df.period_label == df.select(func.max(df.period_label))).show()
>>> "SnowparkPlanException: (1206): Cannot create a Literal for <class 'snowflake.snowpark.dataframe.DataFrame'>"
No longer lazy:
df.filter(df.period_label == subq.first()[0]).show()
Extra syntax:
subq = df.select(func.max(df.period_label).alias("max"))
df.join(subq, df.period_label==subq.max).show()
What is the desired behavior?:
Perhaps we can indicate that the object is meant to be used as a literal via a subquery designation:
df.filter(df.period_label == df.select(func.max(df.period_label)).subquery()).show()
How would this improve snowflake-snowpark-python?
Filtering on the results of a subquery is a very basic pattern. I want to keep execution deferred until the end vs having to pull out the raw value as a literal.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the DataFrame.filter and select APIs, then trace the expression handling behind the reported DataFrame literal error. Define how a subquery designation should preserve lazy execution and how it composes in filter expressions. Done means the demonstrated max-period filter works without collecting the value, with coverage for the proposed subquery usage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sql
- Domain
- data-engineering, database
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100