con / con/mechababs

mechababs aggregate

Open
#123 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
4
Avg merge
15h 39m
Merged PRs (30d)
24

Description

Produce a strawman for yarik to review:
- what data gets collected
- some cases, stuff thats not committed, (babs jobs_status.csv things)
- slurm jobs
- resource usage of slurm jobs (must be collected before slurm drops them from history)
- how that data gets rendered

Concepts to consider:
- cheap render-only mode ie from cache
- query/collect mode ie needs to discover the reality first
- look into git statusd
- where cache lives git-annex uses a special branch to record the "state of the world"
- fscacher might be helpful

Goal: Dashboard "What are we doing, and how done is it"
- list all mechababs campagins
- for a campaign, what studies are involved
- for each study
- what derivatives are complete
- do we need to more info if partially complete?
- record babs specific info, like "has these results branches"?
- these jobs are running/done/pending?
- full list of jobs with resource usage?
- interaction with "ephemeral studies", ie what if the user removes the study from the cluster?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the existing mechababs data sources, including babs jobs_status.csv and Slurm job history, then investigate git statusd, git-annex's special branch, and fscacher. Produce a strawman for yarik covering collection, caching, rendering, campaign/study/job views, resource usage, and ephemeral studies; completion means the proposed design is ready for review.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, python
Domain
data, devtools
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.