Quickwit search errors are not helpful

When searching, if Quickwit runs into an error like an authorization error from S3, you get a very unhelpful error:

{"message":"Internal error: `(Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDQ3YNXVX09ZAH42NB0FBTJ``., split_id: 01GVDQ3YNXVX09ZAH42NB0FBTJ), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDQ7KPN9NHPR5J1GAXNQ259``., split_id: 01GVDQ7KPN9NHPR5J1GAXNQ259), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDQBAS24DHAKK9MQ3B0AN74``., split_id: 01GVDQBAS24DHAKK9MQ3B0AN74), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDQF0218CBFNW7A1ESHKWH8``., split_id: 01GVDQF0218CBFNW7A1ESHKWH8), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDQJNQAXKP5938BGNQG5MER``., split_id: 01GVDQJNQAXKP5938BGNQG5MER), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDQPAFPZVGBVWSX401VZE63``., split_id: 01GVDQPAFPZVGBVWSX401VZE63), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDQT09FZERHC4M4J3102P7T``., split_id: 01GVDQT09FZERHC4M4J3102P7T), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDQXQ0QXHH8130024EXVDHQ``., split_id: 01GVDQXQ0QXHH8130024EXVDHQ), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDR1DESKKSFCE9QAB0DVJ1K``., split_id: 01GVDR1DESKKSFCE9QAB0DVJ1K), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDR6YBC8EZ51PRZP40QS1CM``., split_id: 01GVDR6YBC8EZ51PRZP40QS1CM), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDRAJJP490Q4NVYX0RC9CH2``., split_id: 01GVDRAJJP490Q4NVYX0RC9CH2), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDREEN9RSW5FE1PTXPM6YY5``., split_id: 01GVDREEN9RSW5FE1PTXPM6YY5), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDRKQ0RKBM7PZQ504GNSXBQ``., split_id: 01GVDRKQ0RKBM7PZQ504GNSXBQ), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDRQDF05WQAD9KMG5MMFFZ4``., split_id: 01GVDRQDF05WQAD9KMG5MMFFZ4), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDRV31TWC7SX72MGRKS3TSA``., split_id: 01GVDRV31TWC7SX72MGRKS3TSA), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDS0K6GAHEQT2YR20NPZCD2``., split_id: 01GVDS0K6GAHEQT2YR20NPZCD2), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDS47X6C3ZJ7JJPV008H7MF``., split_id: 01GVDS47X6C3ZJ7JJPV008H7MF), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDS7XY8EDDVDE62MSXTSM8F``., split_id: 01GVDS7XY8EDDVDE62MSXTSM8F), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDSBK1MNK6K66AZ59HBG7BT``., split_id: 01GVDSBK1MNK6K66AZ59HBG7BT), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDSF8XY20C1XAFC7V5FD2NN``., split_id: 01GVDSF8XY20C1XAFC7V5FD2NN), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDSJXNJR85QJ4DE2Z7KG03A``., split_id: 01GVDSJXNJR85QJ4DE2Z7KG03A), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDSPJV8DYE24H1B4208XNBV``., split_id: 01GVDSPJV8DYE24H1B4208XNBV), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDSTBXWKZ8Q6WWSAXRMEXCA``., split_id: 01GVDSTBXWKZ8Q6WWSAXRMEXCA), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDSXYZG755SA0GGDWV7PFDA``., split_id: 01GVDSXYZG755SA0GGDWV7PFDA), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDT1K3AQXKJT8M28JX99XKS``., split_id: 01GVDT1K3AQXKJT8M28JX99XKS), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDT73WBXSV2K0YTNCWKVWEA``., split_id: 01GVDT73WBXSV2K0YTNCWKVWEA), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDTARS5X7NQ8TZAD17EHD1T``., split_id: 01GVDTARS5X7NQ8TZAD17EHD1T), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDTEDX6YHCB3NN1KFZY6Y5J``., split_id: 01GVDTEDX6YHCB3NN1KFZY6Y5J), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDTJ3VGX4PZCPE3EEGDBHA1``., split_id: 01GVDTJ3VGX4PZCPE3EEGDBHA1), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDTNRQR812TDQPSY5SRHCHD``., split_id: 01GVDTNRQR812TDQPSY5SRHCHD), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDTSEGXD5BP40M6QFSXN4M0``., split_id: 01GVDTSEGXD5BP40M6QFSXN4M0), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDTX43Z389VTYDZ2QDFDWZ0``., split_id: 01GVDTX43Z389VTYDZ2QDFDWZ0), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDV0S9P8JVCVAJTV0D4DBAG``., split_id: 01GVDV0S9P8JVCVAJTV0D4DBAG), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDV4EEQ2F3AN5JDKS0WSC1X``., split_id: 01GVDV4EEQ2F3AN5JDKS0WSC1X), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDV83J3QX6BE8DMT0M1VNF0``., split_id: 01GVDV83J3QX6BE8DMT0M1VNF0), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDVBT8K4J3V62RMX11HD6FM``., split_id: 01GVDVBT8K4J3V62RMX11HD6FM), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDVFG1PW8BZMEDYERYYT8M7``., split_id: 01GVDVFG1PW8BZMEDYERYYT8M7), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDVK420ZX6H8G4W02XC8PSY``., split_id: 01GVDVK420ZX6H8G4W02XC8PSY), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDVPT9EC5CRMT8FFGT3JQP0``., split_id: 01GVDVPT9EC5CRMT8FFGT3JQP0), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDVTFWYJJYFV1XZ6JGT7QMZ``., split_id: 01GVDVTFWYJJYFV1XZ6JGT7QMZ), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDVY5KPZX1Y6AMSXJ986TK0``., split_id: 01GVDVY5KPZX1Y6AMSXJ986TK0), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDW1YPSR7ZMFB0SXAKFTF4M``., split_id: 01GVDW1YPSR7ZMFB0SXAKFTF4M), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDW5H7KRZJW4DMVHB34YDW1``., split_id: 01GVDW5H7KRZJW4DMVHB34YDW1), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDW95Z2VT9ZHV627602209B``., split_id: 01GVDW95Z2VT9ZHV627602209B), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDWCVFA9CVGEZZJQ9R9XZF6``., split_id: 01GVDWCVFA9CVGEZZJQ9R9XZF6), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDWGFW056XEYHAZVE38FJ2E``., split_id: 01GVDWGFW056XEYHAZVE38FJ2E), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDWM5M2SM94KC3FXPKCCW4V``., split_id: 01GVDWM5M2SM94KC3FXPKCCW4V), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDWQV6XNMEH425GGHYB9SPA``., split_id: 01GVDWQV6XNMEH425GGHYB9SPA), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDWVGD7Y327KF2N51Y6SPFA``., split_id: 01GVDWVGD7Y327KF2N51Y6SPFA), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDWZ45TB7NXEYE5GZWS22FV``., split_id: 01GVDWZ45TB7NXEYE5GZWS22FV), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDX2VWEWYJDBDM1PT9HZYDM``., split_id: 01GVDX2VWEWYJDBDM1PT9HZYDM), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDX8C9FRDNKK8V83TWT0R06``., split_id: 01GVDX8C9FRDNKK8V83TWT0R06), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDXBZJ67GBX0HR897BP71QM``., split_id: 01GVDXBZJ67GBX0HR897BP71QM), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDXHGMG5XPHCCBNVTE2W2S7``., split_id: 01GVDXHGMG5XPHCCBNVTE2W2S7), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDXN6C8C8HH19M0HDMEYFRC``., split_id: 01GVDXN6C8C8HH19M0HDMEYFRC), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDXRVR2ZAQASS1MMD4D94TX``., split_id: 01GVDXRVR2ZAQASS1MMD4D94TX), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDXW8PB828QJMZKMMYP5T9J``., split_id: 01GVDXW8PB828QJMZKMMYP5T9J), (Internal error: `Failed to fetch hotcache and footer from s3://quickwit/dbl-stats for split `01GVDXWH2WNCZSZW2VZMZ575DH``., split_id: 01GVDXWH2WNCZSZW2VZMZ575DH)`."}

This is also all you get, even in the console logs, for example here it's an authorization error but you would have to guess that from this error.

I imagine this same error occurs for most if not all errors while trying to read splits which could definitely be improved.

The issue names no files, entry points, or tests. Start by reproducing a search failure while reading splits from S3, especially an authorization failure, and trace how the error reaches the search response and console logs. Done means split-read failures expose an actionable underlying cause instead of only the aggregated internal error.