baobabsoluciones / baobabsoluciones/cornflow
Add support for an alternative to Airflow to make minimalistic deployments easy
- Dominant language
- Python
- Stars
- 21
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
I'm always frustrated when I'm trying to deploy cornflow locally. Also, when the tests run for ever in the CI.
**Describe the solution you'd like**
I would like to be able to use Cornflow without Airflow. With a very simple task-runner. Such as directly with Celery + RabbitMQ/ redis. This would make it simpler to test and run locally. While still keeping the airflow deployment for robust/ complete solutions.
**Describe alternatives you've considered**
An alternative is to not have a task runner and run the tasks syncrhonously through http requests in cornflow. But this would be very slow as the client needs to wait for the solution.
**Additional context**
https://github.com/celery/celery
https://docs.celeryq.dev/en/stable/getting-started/introduction.html
Contributor guide
Research direction
No files, tests, or entry points are named. Start by mapping the current Airflow deployment and the CI path that makes local runs and tests slow, then review the linked Celery documentation. Done means Cornflow can run with a simple Celery-based task runner using RabbitMQ or Redis, while the existing Airflow deployment remains available.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rabbitmq, redis
- Domain
- api, backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100