Ensure logging works with Parallel Engines
- 主要语言
- Python
- 星标
- 850
- 派生
- 96
- PR 合并指标
- 30 天内没有已合并 PR
描述
I separated out this issue from #2149:
As posted originally by @freddyaboulton:
Our `raise_error_callback` and `log_error_callback` functions use the logger created by our `get_logger util`.
This is problematic because pipeline evaluation could be happening in parallel so different workers could be attempting to write to the same file at the same time. Moreover, the computation could be happening in a remote machine, so the file that's written to would not be a file accessible by a user. I think the right thing to do is to use to JobLogger introduced in #1975 and add that as an argument to the callbacks.
---
After some discussion in https://github.com/alteryx/evalml/pull/2429#discussion_r658905960, we realized that simply introducing the `JobLogger` as an argument to the callback does not suffice in the case where we raise an error. If we raise an error in the `error_callback` of AutoML, we never get to writing the progress thus far to the log. This is because `JobLogger`'s messages are written to the log on the main node during `_post_evaluation_callback` which will not be hit if we error out beforehand. However, if we choose to write the JobLogger's messages in the callback, the messages will be written on a potentially-remote node that is not accessible to the main process.
The acceptance criteria for closing out this issue is: we need a way to raise an error and have the logs from that node accessible to the main node. The potentially-remote node's log must be accessible to the python code calling evalml.
Freddy had mentioned a few suggestions, including:
1) Not logging to a file and just to stdout. This doesn't feel too great though--as a user, I'd like to have a log of what was executing before my process errored out
2) Have `raise_error_callback` *not* error out (and just write to the JobLogger), then check in AutoMLSearch if we should raise an error. I worry that this may be confusing to users: if they write a custom callback method to raise an error, AutoMLSearch would *not* actually raise the error. Perhaps we could have some logic to check if *any* exception was raised?
贡献指南
评估
这个 Issue 还没有评估数据。