Tombstone dry run - update pipeline to support parallel execution better
- Dominant language
- JavaScript
- Stars
- 23
- Forks
- 62
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
We have used the Prefect ConcurrentTaskRunner for prelim dry runs against the prod data set.
We would like to test if using the DaskTaskRunner will increase throughput by being able to have multiple processes within a given flow.
**Tasks**
- [ ] Update tombstone flow to support usage of DaskTaskRunner. Will probably require refactoring tasks and code in a way where there are no pickling/serialization issues. For example, passing of db connections across tasks will result in serialization issues.
- [ ] Test performance of DaskTaskRunner to verify if it is more performant and able to utilize multiple cores within a given flow and can function with parallel flows
Contributor guide
No contributing guide indexed for this repository
Research direction
No files or tests are named. Start by locating the tombstone flow and its current ConcurrentTaskRunner configuration, then inspect task arguments for database-connection serialization issues. Done means the flow runs with DaskTaskRunner, supports parallel flows, and has a performance comparison for throughput and core utilization.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100