CIROH-UA / CIROH-UA/datastreamcli
Profiling NextGen Research DataStream
- Dominant language
- Python
- Stars
- 3
- Forks
- 6
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 4
Description
Each execution generates a [profiling file](https://ciroh-community-ngen-datastream.s3.amazonaws.com/v2.2/ngen.20250801/short_range/00/VPU_16/datastream-metadata/profile.txt) that logs each of the major datastream steps.
DataStreamCLI Example:
```
DATASTREAM_START: 20250801050130
GET_RESOURCES_START: 20250801050130
GET_RESOURCES_END: 20250801050146
DATASTREAMCONFGEN_START: 20250801050146
DATASTREAMCONFGEN_END: 20250801050223
NGENCONFGEN_START: 20250801050223
NGENCONFGEN_END: 20250801050239
GET_FORCINGS_START: 20250801050239
GET_FORCINGS_END: 20250801050241
VALIDATION_START: 20250801050241
VALIDATION_END: 20250801050314
NGEN_START: 20250801050315
NGEN_END: 20250801050536
MERKLE_START: 20250801050536
MERKLE_END: 20250801050545
TAR_START: 20250801050546
TAR_END: 20250801050549
DATASTREAM_END: 20250801050550
S3_MOVE_START: 20250801050550
S3_MOVE_END: 20250801050553
```
There likely exist some low hanging fruit for speed-ups that will save money at scale. Effort has been put into optimizing forcings processing and validation, but the commands related to the BMI file generation/retrieval are largely un-optimized (DATASTREAMCONFGEN, NGENCONFGEN).
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.