Mars exception not raised

Nobody has claimed this yet.

**Describe the bug**
When mars subpool execute a subtask failed, the exception didn't propagate to mainpool and supervisor

**To Reproduce**
To help us reproducing this bug, please provide information below:
1. Your Python version: python3.7.9
2. The version of Mars you use: mars 0.8.1
3. Versions of crucial packages, such as numpy, scipy and pandas: numpy==1.17.3 scipy==1.5.0 pands==1.3.0
4. Full stack of the error.
```
2022-03-01 13:20:37,096 INFO utils.py:147 -- Note: NumExpr detected 16 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2022-03-01 13:20:40,911 INFO api.py:33 -- Finished initialize the metrics, config is {'metric': {'backend': 'ray'}}, backend is ray
2022-03-01 13:20:43,724 INFO pool.py:329 -- Start to init sub pool.
2022-03-01 13:20:43,730 WARNING patch_import.py:19 -- Patch module lock.
2022-03-01 13:20:43,730 INFO entrypoints.py:18 -- Patch import for debug:
2022-03-01 13:20:43,735 INFO entrypoints.py:20 -- Loading extension: init = dsmp:_init_extension
2022-03-01 13:20:44,412 WARNING odps_op_with_dsmp.py:104 -- inject dsmp to odps op.
2022-03-01 13:20:44,423 INFO api.py:33 -- Finished initialize the metrics, config is {'metric': {'backend': 'ray'}}, backend is ray
2022-03-01 14:02:52,261 INFO processor.py:604 -- Start to run subtask: qne7KJAZozYOBtG8lPx1gqyr on ray://mars_cluster_1646112015/42/8.
2022-03-01 14:02:56,828 ERROR ray.py:212 -- ray.get error Object a09676b3a6d4237e425b08ffe827c3390dbbedea8d01008021000000 is lost due to node failure.
Traceback (most recent call last):
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/storage/ray.py", line 210, in get
return await object_id
ray.exceptions.ObjectLostError: Object a09676b3a6d4237e425b08ffe827c3390dbbedea8d01008021000000 is lost due to node failure.
2022-03-01 14:02:56,829 ERROR threading.py:870 -- Got unhandled error when handling message ('get', 1, ([('HX52gGN5NG9sskVETs0BbiKo', ('e248c2d9fc76474e2737884305fde30e', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('da04dfff5678981bea045a8bc8982a68', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('08517e44227503ce1cdc9a8531a9a502', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('7987e72823a28feea6da1078fff7849e', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('82d6a177ad7189d52c69b526d8c32208', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('eb26a2325d97169f9b176ae487762fcc', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('a1b43c45b8c02a15b4f83ae658da6236', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('d18b227e8d2c0374bb601ba40fde1fdf', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('cd3a799814dbc94d3b485482b3686d39', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('cc99afc84a2861adafca58a25040fab9', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('4411acc8447237fdaa060a3839b7b6ab', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('27cc451495599c3a87283151399debfd', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('432d214162d736d622a4b3a98885fd56', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('59ce82fde0d58ffb71037fd3a3bbdba9', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('7c310b639e9c3c1a879d0570bc6a2ee3', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('734722566891732af606f772ad873049', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('c56af9e57458f9c453d6812967e4e281', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('67a28e08bfd8a28b1724ae6e0e391b6d', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('6848940c6775904df0ae54719200931e', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('478ae75c2562fc45cac1a000c4a351d5', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('87deced2ed53d29645423edb9541a1ae', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('9fc8af9b23a175bdbaf6949c5f3870b6', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('4e39d30ccf1b90fafb7fcd7c02486970', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('3623040fc7d75917a95478f3ed186c9d', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('c81d5542bb2d311d9c5cc1a3b7a7fa42', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('7e3a0376b1de7cd43373ee1752fb6b5d', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('9eb3a23eda012ecb6206a40e819ce93e', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('a25cb6174cac42452f5051b5f45c035c', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('8903298273ca7f74d19a66eaed3838bd', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('b2694f021b2bd9719642900b2ceb115e', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('6d0a32ae56d6cd9a0048f1990eb98352', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('30a0a7b56700f4c7b3b6a378b7d96e0f', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('e38dcff15acbe77dd64711f380ef2807', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('ae12b95c865460c2442e145f0f284c2c', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('a886569d796b06dbafe94dae26866ef9', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('3337cb8c654b9978e0cfa2f815f4ca4e', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('7db4eb54cb4a124935012f9bbbecce7e', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('0fd6c2c4fecb469a285425955531178f', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('31317b8e90727d6c0eaf2457ab2b388d', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('34d604f9164e48b14b921fc382e77950', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('d3b6f7f5a23c51f340b3b926da3a9e49', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('03259d433172689a68ce9be747b769fb', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('1b971b5f9bb54d38a7a2fca85b87f806', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('5b8e35beff5f8f30de7e5e514b64f0b5', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('0f8f08eb4766f92ae3524c76c44d3b65', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('3de8511365322de0ccfc3584a7464dce', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('7b6de32c5785550878cb04692212967a', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('a37cb0cee521cba54f0c9dd7bb1c160e', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('deabacc53ac9633903db152cff9bcab8', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('23bc9f13ac21bba8502cb4945c221912', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('631d0b5c5f914c28d619f85fadb6e7b7', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('d3252d28d4c08d736d4ac66b0e72df30', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('518e23eac691bf5b2b4bbc5190645597', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('c9b135a453eeeeb4b0cb18bae9c2fe96', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('e1eaff190ac07a19a6787c50d897702f', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('93ddfe9734d4ee87f8688d982b14b931', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('f4d1315436975d5e480cf7d9425b0cad', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('77f9c7226787d8905ab0afeb27cf092e', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('6fe79780b8c774a33ea988b38cdcd291', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('06e077c11126659c7fbab460b321f961', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('10f0037dfe108072919e79dea15f3415', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('963a2cca87b058c2cdec82ce4aaec65c', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('2deb25958f19c6229d94cd95dc99beb2', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('a5733c248de0e50e2c336865ebad0143', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('704c3a2784f1bac3c863213943666bea', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('ad64b71a55747c7a5457336acd31187d', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('0515523502a27642353e8d6a09ba1437', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('0812afcad321ea6efa24ecff10607d02', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('cc96e0ddc432592495fb521c8056a415', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('35c524926265b5603b9d9c391b06a6cc', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('5161b141fd3588100f22736eb559a961', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('75b7d6ca2f1e16fff0c0220a19c891a3', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('2456ea84749b00c05c833cbdb6ee7c59', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('1ba429189e6a5e0c07ad536ae21e07e1', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('e3bd9d2e57eab4d45d12815403aef2fb', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('88a2fde176aa3dffb86a448e11c178bb', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('989ac65a0ca3480a22ade9e06bfe5aa7', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('c81e189e62515c7cdaac0643a9c86c4a', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('54d4d2ca987113d1d2c9447d1b22a869', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('8441d1a1eb2deaae5b023d41cfc8557a', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('2278b489395a4709f220b03e482b68b2', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('929d69463cb521b69e3dc4e88aa6839e', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('2a973b2d1e55ac1f3a945ba05c71f678', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('0d239d713144312617ebd44f8998c749', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('1cc7399cf9c10d0517cc96dc1a5dbd20', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('4d1381d2e0d66d1d50cf3423ca22ae04', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('0f2f86082574aa5899cf2a1fe370e013', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('4198ad802bc715801dd7b8757b3f0963', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('eb2490ee455a177799144d26ae928eb8', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('5f656e97820ab43ba406d7edbe6eb232', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('d0428658c6bb5f371a226723485e3547', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('4b6d8cbf3a1508f652a20f24c2434933', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('cf62634796716a8a6b59280d97235438', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('7360240ec796d6e599e4bbe37dd75592', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('d976a4cd01214122fa6bee81d5116486', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('25ca3129c4f8ec10ffccadd62b801691', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('bbf77248a8fce6827bac6f4025fc09bf', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('af4c350e377db1ab1f915a13e49be298', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('24fd1661c2d9682b8762336c9203b772', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('51df4faa3753570073ef6bf3c4cf0e7f', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('3b0a7e00128d2ac29623625522ff7290', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('906e283187000e013f6a619647f28cc4', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('cce8ac2efc8f8b3298bd44a9743afb8a', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('a332b5dc62dedbf6f7e5ddd887589a30', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('270cffcb0180c3f604bb76931d58773a', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('23fed820cd3b38fdde5a40d2712d80ad', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('3f5f520201e56817e54bba07d07de51f', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('3022d646c2fb0ca8e4829e9342b116bd', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('310a0e42548edfedf21718796ecf0d53', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('e0932e7f7623a0d3cf4a29ce0b988ff6', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('665d8e5c0ce438950c12db6980262e94', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('7714d2bf800f30b032ab0da166d18320', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('35c0637257378c67751f3a18576fd22a', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('e6ba64be0292c54c9d2c32baea6a97e9', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('ec63a89c0aa4c3eca2a8527cbd30a47c', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('70f187af29280aa3660a2d61dfe30d85', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('cca10c445867f48ae946f5f4ad307bce', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('36e600d581bb1067bcfa0c679ee73eec', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('d24c4c6ceaad73db2438462d1ddedece', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('8f2a5d48cebd4f847578940c8e7dfea0', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('72f8d304097c2f5b3e0a07046f260883', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('b0959c6e43ea6bcb4647ccff923ad536', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('9fa969742e3ddaf6231d0dae1ece3a4d', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('b8d1bdf0902f31c5887932b034769129', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('822e4eee3b115c4d21a1b7bb86442b8d', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('b29a72039bf4ba906f08fa4eb61f886e', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('5dfbe36a7d66d23f0668bffbdab2fc2b', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('d0bd49dd735dd00f9690a60db8d872d6', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('c2c3c3f5ab681073a7e1abbf6d362a46', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('b995e24a68f1c87da26efc14f754eee6', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('12a53fa20042b35b39e54d3fd09d07e9', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('3a85abeba5145532e6495cee053a042d', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('c9681747189d033a69b3581237ae9ced', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('94d0bc0c4f6cd7cbccdf8d5bdede8fdf', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('b03423c2d545c6ac68a5e61571fd4df1', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('a1d01a8f86dc3a1ff92c1befb86332e3', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('ff267da5b4eb48d0c29532d0ebccf6b6', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('80d2ad05ee7b384a980ea315e98b2655', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('49e09ba1d166a7d04d1256a9a8e96bea', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('dea2629ddf74becfe9cc281e4bee37ed', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('941325fe739c6f197da600167893fb9a', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('7d8a01659c99c254f92d801e31eb0c49', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('6e5fef22a4214e0f0791f5f93f3aeb55', (32, 0))), ('HX52gGN5NG9sskVETs0BbiKo', ('4665fa2288eab39416939088f8a7bb68', (32, 0)))], [{'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}, {'error': 'ignore'}]), {}) in actor b'storage_handler_numa-0' at ray://mars_cluster_1646112015/42/8
Traceback (most recent call last):
File "mars/oscar/core.pyx", line 365, in mars.oscar.core._BaseActor.__on_receive__
File "mars/oscar/core.pyx", line 250, in _handle_actor_result
File "mars/oscar/core.pyx", line 293, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 294, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 299, in mars.oscar.core._BaseActor._run_actor_async_generator
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/services/storage/handler.py", line 149, in batch_get
result = yield self._get_data(data_info, conditions)
File "mars/oscar/core.pyx", line 304, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 250, in _handle_actor_result
File "mars/oscar/core.pyx", line 293, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 294, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 299, in mars.oscar.core._BaseActor._run_actor_async_generator
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/services/storage/handler.py", line 88, in _get_data
res = yield self._clients[data_info.level].get(data_info.object_id)
File "mars/oscar/core.pyx", line 304, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 224, in _handle_actor_result
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/storage/ray.py", line 210, in get
return await object_id
ray.exceptions.ObjectLostError: Object a09676b3a6d4237e425b08ffe827c3390dbbedea8d01008021000000 is lost due to node failure.
2022-03-01 14:02:56,838 ERROR threading.py:870 -- Got unhandled error when handling message ('run', 0, (Subtask(_FIELD_VALUES={'subtask_id': 'qne7KJAZozYOBtG8lPx1gqyr', 'subtask_name': None, 'session_id': 'HX52gGN5NG9sskVETs0BbiKo', 'task_id': 'die4t4nbzk1Gl23ow8yDwjsF', 'chunk_graph': , 'expect_bands': None, 'virtual': False, 'retryable': True, 'priority': (2, 0), 'rerun_time': 0, 'extra_config': None, 'stage_id': 'Wtd4IiToGiMzskHTCuPwmGTu', 'logic_id': '68041442389aa548b7d61fd3f86582f2', 'index': 128, 'parallelism': 144, 'bands_specified': False, 'num_cpus': 1, 'submitted': True}),), {}) in actor b'HX52gGN5NG9sskVETs0BbiKo_subtask_processor' at ray://mars_cluster_1646112015/42/8
Traceback (most recent call last):
File "mars/oscar/core.pyx", line 365, in mars.oscar.core._BaseActor.__on_receive__
File "mars/oscar/core.pyx", line 250, in _handle_actor_result
File "mars/oscar/core.pyx", line 293, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 294, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 299, in mars.oscar.core._BaseActor._run_actor_async_generator
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/services/subtask/worker/processor.py", line 622, in run
result = yield self._running_aio_task
File "mars/oscar/core.pyx", line 304, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 224, in _handle_actor_result
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/oscar/debug.py", line 232, in task_with_ex_logged
return await coro
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/services/subtask/worker/processor.py", line 466, in run
input_keys = await self._load_input_data()
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/services/subtask/worker/processor.py", line 145, in _load_input_data
inputs = await self._storage_api.get.batch(*gets)
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/oscar/batch.py", line 144, in _async_batch
return await self.batch_func(args_list, kwargs_list)
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/services/storage/api/oscar.py", line 101, in batch_get
return await self._storage_handler_ref.get.batch(*gets)
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/context.py", line 186, in send
return self._process_result_message(result)
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/context.py", line 70, in _process_result_message
raise message.as_instanceof_cause()
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/pool.py", line 590, in send
result = await self._run_coro(message.message_id, coro)
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/pool.py", line 343, in _run_coro
return await coro
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/oscar/api.py", line 115, in __on_receive__
return await super().__on_receive__(message)
File "mars/oscar/core.pyx", line 371, in __on_receive__
File "mars/oscar/core.pyx", line 365, in mars.oscar.core._BaseActor.__on_receive__
File "mars/oscar/core.pyx", line 250, in _handle_actor_result
File "mars/oscar/core.pyx", line 293, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 294, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 299, in mars.oscar.core._BaseActor._run_actor_async_generator
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/services/storage/handler.py", line 149, in batch_get
result = yield self._get_data(data_info, conditions)
File "mars/oscar/core.pyx", line 304, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 250, in _handle_actor_result
File "mars/oscar/core.pyx", line 293, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 294, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 299, in mars.oscar.core._BaseActor._run_actor_async_generator
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/services/storage/handler.py", line 88, in _get_data
res = yield self._clients[data_info.level].get(data_info.object_id)
File "mars/oscar/core.pyx", line 304, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 224, in _handle_actor_result
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/storage/ray.py", line 210, in get
return await object_id
mars.oscar.backends.message.ErrorMessage.as_instanceof_cause.._MarsError:
2022-03-01 14:02:56,839 ERROR api.py:115 -- Got unhandled error when handling message ('run_subtask', 0, (Subtask(_FIELD_VALUES={'subtask_id': 'qne7KJAZozYOBtG8lPx1gqyr', 'subtask_name': None, 'session_id': 'HX52gGN5NG9sskVETs0BbiKo', 'task_id': 'die4t4nbzk1Gl23ow8yDwjsF', 'chunk_graph': , 'expect_bands': None, 'virtual': False, 'retryable': True, 'priority': (2, 0), 'rerun_time': 0, 'extra_config': None, 'stage_id': 'Wtd4IiToGiMzskHTCuPwmGTu', 'logic_id': '68041442389aa548b7d61fd3f86582f2', 'index': 128, 'parallelism': 144, 'bands_specified': False, 'num_cpus': 1, 'submitted': True}),), {}) in actor b'slot_numa-0_7_subtask_runner' at ray://mars_cluster_1646112015/42/8
Traceback (most recent call last):
File "mars/oscar/core.pyx", line 365, in mars.oscar.core._BaseActor.__on_receive__
File "mars/oscar/core.pyx", line 250, in _handle_actor_result
File "mars/oscar/core.pyx", line 293, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 294, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 299, in mars.oscar.core._BaseActor._run_actor_async_generator
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/services/subtask/worker/runner.py", line 110, in run_subtask
result = yield self._running_processor.run(subtask)
File "mars/oscar/core.pyx", line 304, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 224, in _handle_actor_result
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/context.py", line 186, in send
return self._process_result_message(result)
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/context.py", line 70, in _process_result_message
raise message.as_instanceof_cause()
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/pool.py", line 590, in send
result = await self._run_coro(message.message_id, coro)
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/pool.py", line 343, in _run_coro
return await coro
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/oscar/api.py", line 115, in __on_receive__
return await super().__on_receive__(message)
File "mars/oscar/core.pyx", line 371, in __on_receive__
File "mars/oscar/core.pyx", line 365, in mars.oscar.core._BaseActor.__on_receive__
File "mars/oscar/core.pyx", line 250, in _handle_actor_result
File "mars/oscar/core.pyx", line 293, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 294, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 299, in mars.oscar.core._BaseActor._run_actor_async_generator
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/services/subtask/worker/processor.py", line 622, in run
result = yield self._running_aio_task
File "mars/oscar/core.pyx", line 304, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 224, in _handle_actor_result
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/oscar/debug.py", line 232, in task_with_ex_logged
return await coro
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/services/subtask/worker/processor.py", line 466, in run
input_keys = await self._load_input_data()
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/services/subtask/worker/processor.py", line 145, in _load_input_data
inputs = await self._storage_api.get.batch(*gets)
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/oscar/batch.py", line 144, in _async_batch
return await self.batch_func(args_list, kwargs_list)
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/services/storage/api/oscar.py", line 101, in batch_get
return await self._storage_handler_ref.get.batch(*gets)
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/context.py", line 186, in send
return self._process_result_message(result)
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/context.py", line 70, in _process_result_message
raise message.as_instanceof_cause()
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/pool.py", line 590, in send
result = await self._run_coro(message.message_id, coro)
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/pool.py", line 343, in _run_coro
return await coro
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/oscar/api.py", line 115, in __on_receive__
return await super().__on_receive__(message)
File "mars/oscar/core.pyx", line 371, in __on_receive__
File "mars/oscar/core.pyx", line 365, in mars.oscar.core._BaseActor.__on_receive__
File "mars/oscar/core.pyx", line 250, in _handle_actor_result
File "mars/oscar/core.pyx", line 293, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 294, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 299, in mars.oscar.core._BaseActor._run_actor_async_generator
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/services/storage/handler.py", line 149, in batch_get
result = yield self._get_data(data_info, conditions)
File "mars/oscar/core.pyx", line 304, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 250, in _handle_actor_result
File "mars/oscar/core.pyx", line 293, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 294, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 299, in mars.oscar.core._BaseActor._run_actor_async_generator
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/services/storage/handler.py", line 88, in _get_data
res = yield self._clients[data_info.level].get(data_info.object_id)
File "mars/oscar/core.pyx", line 304, in mars.oscar.core._BaseActor._run_actor_async_generator
File "mars/oscar/core.pyx", line 224, in _handle_actor_result
File "/home/admin/ray-pack/tmp/job/8d010080/pyenv/lib/python3.7/site-packages/mars/storage/ray.py", line 210, in get
return await object_id
mars.oscar.backends.message.ErrorMessage.as_instanceof_cause.._MarsError:
```

Start with mars/storage/ray.py, where the stack trace shows the Ray object retrieval failure, then trace how the error moves through processor.py and the subpool/supervisor flow. Reproduce a failed subtask on the Ray backend and verify that the original exception reaches the main pool and supervisor instead of becoming an unhandled message.

Written by the indexing model from the issue text.