apache / apache/airflow

Integrate Snowflake Notebook Scheduling with Apache Airflow

Open
#40,167 14 comments 9 reactions 0 assignees View on GitHub
area:providers good first issue kind:feature provider:snowflake
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

Enable scheduling and execution of Snowflake Notebooks from Apache Airflow. This feature would allow users to orchestrate and manage Snowflake Notebooks directly within Airflow, leveraging Airflow's scheduling, dependency management, and monitoring capabilities.

### Use case/motivation

The recent launch of Snowflake Notebooks offers a powerful environment for data engineering, analytics, and machine learning workflows. By integrating Snowflake Notebooks with Airflow, users can automate their data workflows more efficiently. This integration will enable users to:
- Schedule regular execution of Snowflake Notebooks.
- Incorporate Snowflake Notebooks into larger, multi-step data pipelines.
- Use Airflow's monitoring and alerting features to manage the execution of Snowflake Notebooks.
- Reduce the need for manual intervention, thereby increasing productivity and ensuring consistency in data processes.

This feature will greatly benefit data teams who already use Snowflake and Airflow, providing a seamless and automated way to manage their end-to-end data workflows.

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are specified. Start by defining the Snowflake Notebook scheduling and execution interface within Airflow, then establish how scheduling, dependencies, monitoring, and alerting should work; done requires an agreed design and validated end-to-end integration.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.