Incorrect runtimes reported in emissions files
还没有人认领这个 Issue。
- 主要语言
- Python
- 星标
- 1.9k
- 派生
- 323
- 平均合并
- 1 天 12 小时
- 30 天内合并 PR
- 12
描述
- CodeCarbon version: 2.6.0
- Python version: 3.10
- Operating System: Linux (Google Colab)
Description
I was trying to record the emissions generated while fine-tuning the Gemma-7B model on an A100 GPU on Google Colab. While the code ran without any errors, the duration reported by the library in the emissions seems to be incorrect.
In two trials, the total duration is reported as 33.5 seconds and 33.2 seconds, respectively, while the actual durations were closer to 30 minutes. I also recorded the emissions subtask-wise, and the durations for individual subtasks are reported as negative (e.g., -1723972816).
Since I did not face this issue with other notebooks I ran on Google Colab with other models and GPUs, as well as other models on the same GPU, I was wondering where the error came from in this scenario? Thanks!
贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
调研方向
在 Google Colab 中使用 CodeCarbon 2.6.0、Gemma-7B 微调和 A100 GPU 重现该报告,并将总时长和子任务时长与实际运行时间进行比较。首先跟踪排放文件和子任务的运行时间值是如何记录的。完成的标准是时长为非负值,并且与观察到的执行时间高度匹配。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- python
- 领域
- backend
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 需要澄清
- 新手友好度
- 30/100