MetOffice / MetOffice/fab

Reduce inodes count

Open
#440 2 comments 0 reactions 0 assignees View on GitHub
discussion question
Dominant language
Python
Stars
9
Forks
9
Avg merge
12d 12h
Merged PRs (30d)
2

Description

In our CI pipeline we are constantly running out of inodes (if a CI run fails, there is no cleanup). I expect that Fab will put additional pressure on the number of inodes, since for each (Fortran) input file, it will create:

1. a preprocessed file (even if the input file is .f90, it will create a copy so that the whole build is done from `build_dir` only)
2. an `.an` file
3. two `.mod` files (one with hash, one without)
4. a `.o` file

Though it also does save by only compiling the absolute minimum number of files. A rough analysis of one of my gungho builds:
```
930 source files
4676 build files

1156 *90 files (note that PSyclone will create two Fortran files for one X90 input file)
1675 *mod files (that's two per *90 file - hashed and non-hased version. Not all *90 files will probably be used in gungho, so that's why we have only around 837.5 files ... and no idea about why this is an odd number, best guess would be left-over mod files, since I rebuild in the same directory, and my fab development version sometimes crashes before cleanup??)
820 *o files
1023 .an files (analysis files)
```
So, of the 5606 overall files, the .hashed mod file and the .an file might not need to be written, saving around 1840 inodes, or around 33% (Fab uses internally cached information to get the analysis information from the analysis step to the compilation step, so writing the files is not really required in a successful build).

Of course, if the files are not written, and a run crashes, important information might be missing.

I am just raising this as a question (with no intention of working on it in the near future :) ): is a reduction of inodes count important for other sites?

There are also different (though more complex to implement) ways of reducing this, e.g. all .an files could be stores in a single file (sorting out concurrent writes from sub processes etc etc).

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the CI pipeline and Fab's handling of the listed preprocessed, .an, .mod, and .o files. Determine which artifacts are required after successful and failed builds, including concurrent subprocess behavior. Done would mean a concrete, agreed approach to reducing inode use without losing necessary recovery information.

Written by the indexing model from the issue text.

Assessment

Tech stack
fortran, python
Domain
build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.