baobabsoluciones / baobabsoluciones/cornflow

Add support for an alternative to Airflow to make minimalistic deployments easy

Open
#581 0 comments 0 reactions 0 assignees View on GitHub
triage
Dominant language
Python
Stars
21
Forks
5
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**
I'm always frustrated when I'm trying to deploy cornflow locally. Also, when the tests run for ever in the CI.

**Describe the solution you'd like**
I would like to be able to use Cornflow without Airflow. With a very simple task-runner. Such as directly with Celery + RabbitMQ/ redis. This would make it simpler to test and run locally. While still keeping the airflow deployment for robust/ complete solutions.

**Describe alternatives you've considered**
An alternative is to not have a task runner and run the tasks syncrhonously through http requests in cornflow. But this would be very slow as the client needs to wait for the solution.

**Additional context**
https://github.com/celery/celery
https://docs.celeryq.dev/en/stable/getting-started/introduction.html

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by mapping the current Airflow deployment and the CI path that makes local runs and tests slow, then review the linked Celery documentation. Done means Cornflow can run with a simple Celery-based task runner using RabbitMQ or Redis, while the existing Airflow deployment remains available.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rabbitmq, redis
Domain
api, backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.